Virtual assistant system to enable actionable messaging

ABSTRACT

A virtual assistant system includes a mobile device to receive an input command corresponding to a function to be performed at one or more external services, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom, and an assistant server configured to receive the transmitted semantic atom, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the external services and configured to generate a proprietary language command corresponding to the received semantic atom for each external service in which the function is to be performed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No.62/090,786, filed on Dec. 11, 2014, in the United States Patent andTrademark Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND OF THE INVENTIVE CONCEPT

1. Field of the Invention

The present general inventive concept relates to a virtual assistantsystem to enable actionable messaging, which can enable control andwireless communications between a plurality of external services, aswell as enable sharing of information between the plurality of externalservices, and to selectively allow sharing of the control with otherusers.

2. Description of the Related Art

Advancements in wireless communications have allowed innovation in datasharing to grow at a rapid rate. A user may use a mobile device (suchas, for example, a phone, tablet computer, wristwatch, etc.) to open upvarious applications/programs, operate various devices, and communicatewith other mobile devices. A development referred to as the Internet ofThings, or IoT, reflects a growing trend of connecting to andcontrolling various services wirelessly.

Users of mobile devices, e.g., smartphones, customarily access a widevariety of services, for example “apps” on their mobile devices. Usersmust often switch from one service to another in order to access and useeach service. Furthermore, the mobile device is limited to requiring theuser to input commands, instructions, and data in a manner unique toeach specific application/program and external service. Morespecifically, programs such as Facebook™ and LinkedIn™ each communicatein their own proprietary languages. If a user wishes to look up aperson's profile in LinkedIn™ after having found that person inFacebook™, for example, the user is required to type in proper searchcriteria in proper fields within the LinkedIn™ program itself. As such,although a user may separately communicate with Facebook™ and LinkedIn™,these programs do not and cannot communicate with each other.

As another example, Phillips has developed a light emitting diode (LED)light bulb (i.e., the Phillips “Hue”) that allows a user to changecolors emitting therefrom via a mobile device that is running anapplication associated with the bulb. More specifically, the user mustdownload and install, onto the mobile device, the application associatedwith the bulb, which communicates with the bulb using the Phillips Hueproprietary language. After the application is installed and running onthe mobile device, the user may then control the tone, contrast, andcolor of the bulb using the mobile device. However, the user cannotgrant access to the light bulb to another user operating another mobiledevice. Instead, if the other user wishes to control the light bulb withthe other mobile device, the other user must also download and installthe application associated with the bulb into the other mobile device.

Moreover, the user cannot use information within the applicationassociated with the light bulb to interact with another application. Forexample, the user may not use information within the light bulbapplication to purchase more light bulbs in another application such asAmazon™. As such, the application associated with the bulb is limitedmerely to allowing a user to control the particular light bulbassociated with the application stored within the user's mobile device.

Further, Apple™ and Google™ each include a voice assistant (Siri™ forApple and Google Now™ for Google™) on its respective mobile device thattranslates a voice received into a search program. However, the Siri™and Google Now™ assistants are limited to providing a search and somedevice actions, and have a limited integration with other applicationssuch as OpenTable™, etc.

Accordingly, there is a need for a system that allows all servicesconnected with the system to communicate therebetween.

Further, there is a need to be able to control any external servicesdesired wirelessly, for example via the web, by inputting commands toperform such controls from a hand-held device.

SUMMARY OF THE INVENTIVE CONCEPT

The present general inventive concept provides a mobile device,assistant server, and virtual assistant system configured to allowcommunication between a plurality of services, and control thereof.

Additional features and utilities of the present general inventiveconcept will be set forth in part in the description which follows and,in part, will be obvious from the description, or may be learned bypractice of the general inventive concept.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a virtual assistantsystem including a mobile device to receive an input commandcorresponding to a function to be performed at one or more externalservices, to translate the input command into a semantic atomrepresenting the command, and to transmit the semantic atom, and anassistant server configured to receive the transmitted semantic atom,the assistant server including a plurality of plugins, each plugincorresponding to a respective one of the external services andconfigured to generate a proprietary language command corresponding tothe received semantic atom for each external service in which thefunction is to be performed.

The assistant server may be further configured to receive output datafrom the one or more external services as a result of performing thefunction, to convert the received output data to one or more semanticatoms representing the received output data, and to send the one or moresemantic atoms representing the received output data to at least one ofthe mobile device and another plugin corresponding to another externalservice.

The assistant server may be further configured to receive output datafrom the one or more external services at the corresponding one or moreplugins as a result of performing the function, and to add the receivedoutput data to the semantic atom.

The assistant server may be configured to combine a plurality ofsemantic atoms to create a single merged semantic atom.

The assistant server may be configured to separate a semantic atom intoa plurality of semantic atoms, each of the plurality of semantic atomsincluding a subset of information contained in the original semanticatom.

The assistant server may be configured to generate one or more copies ofone or more semantic atoms.

The assistant server may be further configured to generate one or morecopies of a semantic atom representing an input command, and to providethe one or more copies of the semantic atom to one or more plugins togenerate proprietary language commands for the corresponding externalservices to perform functions corresponding to the input command.

At least one of the mobile device and the assistant server may beconfigured to transmit one or more semantic atoms to at least one othermobile device.

The one or more semantic atoms transmitted to the at least one othermobile device may represent user identification credentials to authorizea function at an external service.

The one or more semantic atoms transmitted to the at least one othermobile device represent output data received from the one or moreexternal services.

The assistant server may be further configured to delete the semanticatom after generating the proprietary language command when the sematicatom is determined to no longer be required for further informationtransmissions thereof with other services or the mobile device.

The assistant server may be further configured to store the semanticatom after generating the proprietary language command.

The assistant server may provide the semantic atom representing theinput command to a plurality of the plugins to generate proprietarylanguage commands for each of the corresponding plurality of externalservices when the input command corresponds to a function or functionsto be performed at the plurality of external services.

The assistant server may store credentials including data to authorizethe mobile device to perform input commands for the one or more externalservices.

An input command at the mobile device may extend authority to performinput commands for the one or more external services to at least oneother mobile device.

Another input command at the mobile device may revoke the authorizationof the at least one other mobile device to perform input commands forthe one or more external services.

Extending authority to perform input commands at the one or moreexternal services to the at least one other mobile device may includecopying the credentials and the plugin corresponding to the one or moreexternal services to an assistant server accessed by the other mobiledevice.

The input command to extend authorization to the at least one othermobile device to perform input commands to the one or more externalservices may include information to limit the authorization to apredetermined level of control, a predetermined location in whichcontrol is permitted, a predetermined time frame in which the control ispermitted and a predetermined access path in which the other userapproaches the one or more external services.

The assistant server may analyze context information of the inputcommand before converting the input command to a semantic atom toaccurately determine the intended function to be performed by the inputcommand.

The context information analyzed by the assistant server may include atleast one of a location of the mobile device when the command is input,a time of day when the command is input, co-presence of specificindividuals, involuntary actions by a user, and action sequences takenby a user of the mobile device.

The action sequences taken by the user may include a plurality of inputcommands made simultaneously or sequentially.

The action sequences taken by the user may include an input command thatis invalidated and followed by another input command.

The assistant server may include a plurality of server devices, theplurality of server devices being configured to transfer semantic atomsbetween each other.

The plurality of server devices may include a main server deviceconfigured to communicate with the mobile device, and one or moresubordinate server devices configured to communicate with the mainserver device.

The plurality of server devices may transmit semantic atoms to andreceive semantic atoms from a plurality of mobile devices.

Each of the plurality of server devices may be in communication with acorresponding mobile device.

Each of the plurality of server devices may be in communication with atleast one other of the plurality of server devices.

One or more of the plurality of server devices stores data regarding auser of the virtual assistant system.

The one or more server devices storing data regarding the user of thevirtual assistant system may be configured to be disconnected from theplurality of server devices in response to an input command from theuser.

One or more of the server devices may store one or more plugins, each ofthe one or more plugins corresponding to a different external service,and each of the server devices, upon receiving a semantic atomrepresenting an input command corresponding to a function to beperformed at an external service, may transmit the semantic atom to theserver device storing the plugin corresponding to the external service.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a method of controllingexternal services, the method including receiving an input command at amobile device, the input command corresponding to a function to beperformed at a selected one of a plurality of external services,translating the input command into one or more semantic atoms at themobile device, transmitting the one or more semantic atoms to anassistant server configured to manage semantic atoms, providing the oneor more semantic atoms to a plugin corresponding to the selectedexternal service to generate a proprietary language command for theselected external service, and controlling the selected external serviceto perform the function by transmitting the obtained proprietarylanguage command to the selected external service.

The method may further include providing the one or more semantic atomsrepresenting the received output data to one or more pluginscorresponding to one or more other selected external services togenerate proprietary language commands corresponding to the receivedoutput data, and controlling the one or more other selected externalservices to perform functions by transmitting the proprietary languagecommands to the one or more other selected external services.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a non-transitorycomputer-readable medium containing computer-readable codes to performthe method of controlling external services.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a server device,including a storage system to store semantic atoms which can be sharedbetween a plurality of external services, the storage system furtherincluding a library of commands to perform functions at the plurality ofexternal services, a plurality of plugins, each plugin corresponding toa respective one of the plurality of external services, each pluginbeing configured to translate between semantic atoms and a respectiveproprietary language of the corresponding external service, and aplatform configured to share semantic atoms between the plurality ofexternal services by using the proprietary language translated at therespective plugins and to share semantic atoms with a mobile device inwhich commands to perform a function are initially input and transmittedto the platform as sematic atoms.

The storage system may store at least one command set corresponding to arespective one of the plurality of external services, each command setincluding commands to perform corresponding functions at the respectiveexternal service.

Each plugin may be created at the server device to include a set ofcommands selected from the stored library of commands, the set ofcommands to correspond with associated functions to be performed at thecorresponding service.

The platform may be configured to wirelessly connect to a plurality ofmobile devices to share information therebetween and to receive inputcommands therefrom, the information being shared and input commandsbeing received being provided as semantic atoms.

The foregoing and/or other features and utilities of the present generalinventive concept may be achieved by providing a mobile device,including a sensor to receive an input command to perform a function atone or more external services, and an assistant interface to translatethe received input command into a semantic atom and to send the semanticatom to an external server to perform the function at the one or moreexternal services via a corresponding plugin or to receive a semanticatom including proprietary language of the one or more external servicesto directly perform the function at the one or more external services.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other features and utilities of the present generalinventive concept will become apparent and more readily appreciated fromthe following description of the embodiments, taken in conjunction withthe accompanying drawings of which:

FIG. 1A is a block diagram illustrating a virtual assistant systemaccording to an exemplary embodiment of the present general inventiveconcept;

FIG. 1B is a block diagram illustrating a virtual assistant systemaccording to another exemplary embodiment of the present generalinventive concept;

FIG. 2A is a block diagram illustrating messaging between mobile devicesaccording to an exemplary embodiment of the present general inventiveconcept;

FIG. 2B is a block diagram illustrating messaging between mobile devicesand one or more external services according to an exemplary embodimentof the present general inventive concept;

FIG. 3A is a block diagram illustrating merging semantic atoms accordingto an exemplary embodiment of the present general inventive concept;

FIG. 3B is a block diagram illustrating nesting semantic atoms accordingto an exemplary embodiment of the present general inventive concept;

FIG. 4A is a block diagram illustrating an exemplary embodiment of amobile electronic device usable with the virtual assistant systemsillustrated in FIGS. 1A-1B;

FIG. 4B is a block diagram illustrating another exemplary embodiment ofa mobile electronic device;

FIG. 5A is a block diagram illustrating an exemplary embodiment of thesensor part of the mobile electronic device of FIG. 4A;

FIG. 5B is a block diagram illustrating an exemplary embodiment ofexternal input sensors and devices usable with the mobile electronicdevice of FIGS. 4A and 4B;

FIG. 6A is a block diagram illustrating an exemplary embodiment of anassistant server usable with virtual assistant systems illustrated inFIGS. 1A-1B;

FIGS. 6B and 6C are block diagrams illustrating multiple instances ofthe assistant server according to exemplary embodiments of the presentgeneral inventive concept;

FIGS. 7A and 7B are flowcharts illustrating command processes of thevirtual assistant system, according to exemplary embodiments of thepresent general inventive concept;

FIG. 8 is a block diagram illustrating the flow of operations of thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept;

FIG. 9 is a block diagram illustrating a process of using semantic atomsto communicate between external services according to an exemplaryembodiment of the present general inventive concept;

FIG. 10 is a block diagram illustrating a process of cross-user sharingof semantic atoms according to an exemplary embodiment of the presentgeneral inventive concept;

FIG. 11 illustrates a process of adaptive addition of a conscious inputcommand to a library of commands, according to an exemplary embodimentof the present general inventive concept;

FIG. 12 illustrates a process of adaptive additions of unconscious inputcommands to a library of commands, according to an exemplary embodimentof the present general inventive concept;

FIG. 13 illustrates a process of consolidating a plurality of gestureinputs in a gesture library to a single gesture input, according to anexemplary embodiment of the present general inventive concept;

FIG. 14 is a diagram illustrating conditional access to the virtualassistant system, according to an exemplary embodiment of the presentgeneral inventive concept;

FIG. 15 is a diagram illustrating access privilege sharing of thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept;

FIG. 16 is a diagram illustrating access privilege sharing betweenassistant servers, according to an exemplary embodiment of the presentgeneral inventive concept;

FIG. 17 is a diagram illustrating authority sharing and revocation ofthe virtual assistant system, according to an exemplary embodiment ofthe present general inventive concept; and

FIG. 18 is a diagram illustrating mass-addition of services to thevirtual assistant system, according to an exemplary embodiment of thepresent general inventive concept.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to various exemplary embodiments ofthe present general inventive concept, examples of which are illustratedin the accompanying drawings, wherein like reference numerals refer tothe like elements throughout. The embodiments are described below inorder to explain the present general inventive concept while referringto the figures. Also, while describing the various exemplary embodimentsof the present general inventive concept, detailed descriptions aboutrelated well-known functions or configurations that may diminish theclarity of the points of the present general inventive concept will beomitted for brevity of the description.

It will be understood that although the terms “first” and “second” areused herein to describe various elements, these elements should not belimited by these terms. These terms are only used to distinguish oneelement from another element. Thus, a first element could be termed asecond element, and similarly, a second element may be termed a firstelement without departing from the teachings of this disclosure.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

All terms including descriptive or technical terms which are used hereinshould be construed as having meanings that are obvious to one ofordinary skill in the art. However, the terms may have differentmeanings according to an intention of one of ordinary skill in the art,case precedents, or the appearance of new technologies. Also, some termsmay be arbitrarily selected by the applicant, and in this case, themeaning of the selected terms will be described in detail in thedetailed description of the invention. Thus, the terms used herein haveto be defined based on the meaning of the terms together with thedescription throughout the specification.

Also, when a part “includes” or “comprises” an element, unless there isa particular description contrary thereto, the part can further includeother elements, not excluding the other elements. In the followingdescription, terms such as “unit” and “module” indicate a unit toprocess at least one function or operation, wherein the unit and theblock may be embodied as hardware or software or embodied by combininghardware and software.

Hereinafter, one or more exemplary embodiments of the present generalinventive concept will be described in detail with reference toaccompanying drawings.

FIG. 1A illustrates a virtual assistant system 10 according to anexemplary embodiment of the present general inventive concept, and FIG.1B illustrates a virtual assistant system 10 according to anotherexemplary embodiment of the present general inventive concept.

The virtual assistant system 10 according to the exemplary embodimentsof FIGS. 1A and 1B may include a mobile electronic device 100, anassistant server 200, and an external service 300 (or services 300 a-n).

The virtual assistant system 10 is configured to enable actionablemessaging. “Actionable messaging” as used herein means messaging whichmay be used to control external services 300 a-n. Data may be messagedboth to and from one user (e.g. from mobile device 100) to another user,as well as to and from one user and one or more external services 300a-n. Messaging in this manner enables one or more users of the virtualassistant system 10 to issue commands to one or more external services300 a-n, in a manner to be described infra.

“Messaging” as used herein is distinct from a “feed.” A “feed” iseffectively a stream of data, which a user may access to be providedwith the data available at that moment in the stream. A user maycontribute to the stream, e.g. by providing data, but there is limitedback-and-forth communication. “Messaging,” on the other hand, is moreinteractive, in which one or more specific users (or service(s) 300) aretargeted to receive one or more specifically selected messages. Thetargeted user(s) (or service(s) 300) may respond to the message, e.g.with an acknowledgment, an output from an action performed in responseto the message, additional information, and so on. Put another way,messaging encourages two-way conversations, whereas feeds aresubstantially one-way communication.

The mobile electronic device 100 (hereinafter “mobile device”) mayinclude capabilities of wireless or wired connectivity with the Internetand various networks in which the mobile device 100 can be configured tocommunicate. More specifically, the mobile device 100 may communicatewith the assistant server 200 and/or the external service(s) 300 a-nusing Wi-Fi, Wi-Fi direct, Near-field communication (NFC), Bluetooth,radio frequency identification (RFID), Ethernet, FireWire, universalserial bus (USB), high definition multimedia interface (HDMI), or anyother type of wireless or wired connection, but is not limited thereto.The communications between the mobile device 100, the assistant server200, and the external service(s) 300 may be performed via the Internet,including cloud computing applications, satellite, a local network, etc.

The mobile device 100 may be any type of computing platform, including asmartphone, a tablet computer, a laptop computer, a smart watch, or awearable device, but is not limited thereto, and may even includenon-mobile devices such as a desktop computer or other stationarycomputing devices. The mobile device 100 is described in greater detailbelow with respect to FIG. 4A.

The mobile device 100 may also include an assistant interface 110 whichis specifically created as part of the virtual assistant system 10 toallow all actions, commands, etc., originating at the mobile device 100,or obtained by the mobile device 100 via another sensor-type device, tobe communicated to the assistant server 200, and vice versa, to allowthe assistant server 200 to communicate necessary information to themobile device 100 in which the assistant interface 110 is provided. Theassistant interface 110 also allows communications with other assistantinterfaces 110 a-n provided on one or more other users' respectivemobile devices 100 a-n, or other servers which may contain libraries ofcommands, to be described in more detail infra. The assistant interface110 provided at the mobile device 100 can perform similar operations asa “plugin” or “connector,” among other operations to be describedherein. The assistant interface 110 is described in greater detail belowwith respect to FIGS. 2A, and 5A-5B. The assistant interface 110 can beprogrammed into a mobile device 100 by a program designer or provided inthe form of an application which may be downloaded to the user's mobiledevice 100 from an external source. The assistant interface 110 may alsobe called the “Client,” but is not limited thereto. A user may connecttheir mobile device 100 to the assistant server 200 by programming ordownloading and installing the assistant interface 110 into the mobiledevice 100.

Alternatively, as illustrated in FIG. 4B, the assistant interface 110may be provided on the same hardware installation as the assistantserver 200. In other words, the assistant server 200 may be provided atthe mobile device 100 together with the assistant interface 110, inwhich case the assistant interface 110 is not required to wirelesslycommunicate with a backend-type server system in order to understand theactions required to take in order to perform any functions at a service300. Thus, in the exemplary embodiment as illustrated in FIG. 4B, to bediscussed in more detail below, the virtual assistant system 10 may beentirely self-contained in a mobile type device 100 as defined above.

The assistant server 200 can be a server that may include, for example,a computer, a dedicated computer (i.e., a back-end server),cloud-computing technology, or a plurality of computers, running one ormore application(s) (i.e., software) that allows communication with themobile device 100 (together with the assistant interface 110) via theInternet, and also allows communication with an infinite range ofexternal services 300 a-n (e.g. a television, a mobile deviceapplication, etc.) via the Internet. The assistant server 200 may storevarious plugins 220 a-n, as well as command inputs, e.g., languages,icons, gestures, sensor-commands, programs, applications, commands, andlibraries including any combinations of the above items, which areusable by the mobile device 100 running the Client 110. The assistantserver 200 described here and illustrated in FIGS. 1A, 1B, and 6A has anopen type platform 210, i.e., a facility whereby any software engineer(also referred to as the software or plugin developer) may create aconnector (a “plugin” 220) therein to correspond with any service(s)300. Each plugin 220 corresponds to a different external service 300.The assistant server platform 210 and plugins 220 may be included alltogether in one hardware configuration of the assistant server 200.

Several assistant servers 200 may communicate with one another to sharefunctionality, thereby effectively operating as a single assistantserver 200 spread out over multiple other servers. This arrangement, tobe described in greater detail infra with reference to FIGS. 6B and 6C,is referred to herein as “multiple instances of the assistant server.”

The external services 300 a-n correspond to anything which may becontrolled (perform functions) in response to command inputs from one ormore users. The external services 300 a-n may include, for example,devices such as lamps or doors, as well as software programs andapplications or “apps” that a user might have on their computer orphone, but are not limited thereto. The external service(s) 300 a-n mayinclude any type of device or service that is connectable to the mobiledevice 100 and/or the assistant server 200 via a local network or theInternet, or other wireless forms of communication, as described above.Examples of services 300 a-n for which plugins 220 may be createdinclude home and office systems (e.g., security, lighting, thermostats),automotive devices, apps, wearable devices, online services, web pages,etc. This list of services 300 a-n is provided herein as examples only,and are far more expansive to millions of other services.

Services 300 a-n may be accessible via the Internet, for example, aswell as the mobile devices 100 a-n themselves. Services 300 a-n whichare not immediately capable of wireless communications (for example,lamps, garage doors, etc.) are also capable of being Internet-accessibleor otherwise capable of communicating with other apparatuses/devices,including the mobile device 100 and assistant server 200, by beingsupplied with a wireless chip or other device, such as, for example,GoGogate™ for garage doors. Normally inaccessible services 300 a-n maybe made wireless-ready by connecting them to an Internet-accessibledevice. This concept has been recognized and is currently being referredto in discussions relating to “the internet of things,” also referred toas IoT. Such services 300 a-n are also accessible via a local networktype of communication so that the mobile device 100 may ultimatelycontrol the services 300 a-n directly once the proprietary language,including necessary credentials, is obtained by the mobile device 100.

It is well known that each of the external services 300 a-n usedthroughout the world have their own corresponding proprietary datastructures in their own proprietary languages, as well as specificcredentials which are required to control/operate the respectiveexternal service 300. However, the plugins 220 act as translators (orinterpreters) between the proprietary data structures and the semanticlanguage. Each plugin 220 therefore allows the assistant server 200 tocommunicate with the respective external service 300 in its respectiveproprietary language via the Internet, WiFi, Bluetooth, WiFi direct,HDMI, USB, FireWire, NFC or other means in order to execute a user'swishes, either explicit or inferred by the virtual assistant system 10.The plugin 220 also can provide the assistant server 200 with anyproprietary language information regarding the corresponding externalservice 300, including specific credentials, which may be necessary inorder to permit security access to the desired service 300 to becontrolled.

The various plugins 220 a-n correspond to various respective externalservices 300 a-n (including, for example, devices, services, programs,and applications). An engineer/developer may create such a plugin 220 atthe assistant server 200 to allow the assistant server 200 tocommunicate with the corresponding external service 300 in itsrespective proprietary language. The engineer/developer may create theplugin 220 for the corresponding service 300 based on specifications ofthe external service 300. Accordingly, an infinite number of plugins 220a-n can be created at the assistant server 200 to allow one or moreusers to control the various respective services 300 a-n in the manneras described above. Furthermore, although the plugins 220 a-n areillustrated as being programmed/stored at the assistant server 200, oneor more plugins 220 a-n may also be stored at the mobile device 100 (forexample, programmed/stored in the assistant interface 110), therebyallowing the assistant interface 110 to communicate directly with therespective external services 300 a-n.

The plugins 220 can be programmed in any programming language, such as,for example, C, Python, Java, C++, OCAML, etc., but are not limitedthereto. Furthermore, the plugins 220 a-n can also be programmed using agraphical interface or web form that allows a programmer to makeselections of visual data (such as icons, pictures, text, etc.) in orderto allow the plugins 220 a-n to be generated without using anyprogramming language at all (i.e. the code is generated automaticallyfrom the insertion and/or incorporation of the visual data into thegraphical interface or web form).

The combination of the mobile device 100 running the Client 110 and theassistant server 200 is referred to herein as the “virtual assistant”but is not limited thereto. Alternatively, users may become accustomedto referring to the mobile device 100 as their own personal “virtualassistant,” since a consumer will not necessarily be aware of thecomponents and processes involved with the operations and communicationsinvolving the mobile device 100 and the assistant server 200. In otherwords, the user may only perceive interacting with a mobile device 100to control external service(s) 300.

The assistant server 200 also has a designated storage 250 (illustratedin FIG. 6A) associated with it. The storage 250 may be included as partof the assistant server 200 (for example, if the assistant server 200 isa back-end server), or the storage 250 may be an external storage whichthe assistant server 200 can access, such as, for example, an externalserver, a remote storage device or a cloud storage service. A softwareengineer may select from a predefined library of command inputs whichare stored in the designated storage 250 in order to define a commandset 240 (illustrated in FIG. 6A) which allows the user to presentcorresponding commands to each external service 300 a-n via theassistant server 200. The library of command inputs may include anyinputs which may be received by the mobile device 100. The commandinputs may include, for example, gestures and icons, as well as audibleor voice commands (stored as actionable data), and other forms ofcommands, which are stored in the storage 250 (to be described in moredetail below with respect to FIG. 6A) associated with the assistantserver 200, to define a command set 240 which allows the user at themobile device 100 to present these selected commands to the respectiveexternal services 300 a-n via the assistant server 200. In other words,the storage 250 stores a vast vocabulary of commands, such as, forexample, gestures, icons, voice data, etc., in the form of a commonvocabulary format which can be selected from by the software engineer inorder to create separate command sets 240 a-n associated with each ofthe external services 300 a-n. These commands are selected to createcommand sets 240 a-n to correspond with respective functions that can beperformed at each of the external services 300 a-n. The commands storedin the storage 250 can also be shared between an infinite number of theexternal services 300 a-n and mobile devices 100 a-n. As used herein,“command inputs” or “commands” may include any form of input to controlan action of the virtual assistant system 10. It will be understood thatthe specific command inputs used in the examples infra may be replacedby other command inputs. For example, if a gesture is described, thiscommand input of a gesture may be replaced by a command input of, e.g.,touching a displayed icon, depending on the particular exemplaryembodiment of the present general inventive concept.

As will be described below with respect to FIG. 1B, under certainpredefined situations, these specific commands from the defined commandset 240 can be presented to the corresponding external service 300directly from the mobile device 100 through the assistant interface 110in a local control mode. The assistant server 200 also providesassistance in this local mode, as will be described in more detailinfra.

As pointed out above, the selections of commands by the plugin developercan be grouped together and saved in the storage 250 as separate commandsets 240 a-n, which can be specific to the corresponding services 300a-n (illustrated in FIG. 6A). In other words, storage 250 can beconfigured with a vast vocabulary of gestures, icons, voice data, andother type commands provided in a common vocabulary format, which can beshared between all services 300 a-n which are connected to the assistantserver 200. A software engineer may select from this common vocabularyof commands while creating a plugin 220 in order to create a command set240 to be associated with the various functions or operations that thecorresponding external service 300 can perform. This common vocabularyformat of commands will be part of a language hereinafter referred to asa “semantic language.” from which. The semantic language is used torepresent objects and concepts, a list which may include but is notlimited to commands, authority to issue commands, people, and objectsand services (e.g., songs, movies, restaurants, weather reports, Wi-Fiaccess, etc.).

The semantic language used at the assistant server 200 allows theassistant interface 110, provided within the mobile device 100, tocommunicate the many gestures, icon touches, voice data, etc., from themobile device 100 (in raw input form) to the assistant server 200 sincethe assistant interface 110 translates the received gestures, icontouches, voice data, etc. into this created semantic language in orderto communicate all types of information relevant to different services300 a-n therebetween via a single language within the assistant server200.

The semantic language allows the assistant server 200 to function as auniversal translator between the various services 300 a-n. Morespecifically, the semantic language can be used for communicationbetween the assistant server platform 210 and each of the plugins 220a-n for all of the services 300 a-n for which a plugin 220 is created atthe assistant server 200.

Various entities and concepts may be represented as units of data in thesemantic language. These units of data are referred to herein as“semantic atoms” or “atoms” 1000. A semantic atom 1000 may representanything in the world, independent of any particular external service300. Semantic atoms 1000 may represent data, for example a name or atime, and may also represent abstract concepts and other information.For example, a semantic atom 1000 could represent a command, a person, abusiness, a movie, a location, a WiFi access point, a song, a Socialmedia post, a light bulb, a car, a GPS device, a virtual assistant, etc.

The virtual assistant system 10 uses these semantic atoms 1000 to allowvarious services 300 a-n to communicate seamlessly with one another, aswell as with the assistant server 200 and the mobile device 100 (andother mobile devices 100 a-n). The various services 300 a-n, operatingin conjunction with plugins 220 a-n, may utilize one or more of thesemantic atoms 1000 a-n to perform their respective functions.

The system of semantic atoms 1000 is maintained in the assistant server200. The exact definitions of, e.g., command inputs, can evolve overtime, including with the participation of an open developer community.This evolution with an open developer community is an approach known as‘Folksonomy,’ when applied to taxonomies which evolve through communityparticipation.

The applications of semantic atoms 1000, and the number of things whichmay be represented by semantic atoms 1000, are infinite. Semantic atoms1000 can be represented in a variety of formats, including text, YAML,XML and other such languages. These formats may correspond to thesemantic language, which as described above is a standardized languageor set of languages in which semantic atoms 1000 are represented.Semantic atoms 1000 may be stored in the mobile device 100 (for example,in storage unit 140, illustrated in FIG. 4A), or in the assistant server200 (for example, in storage 250, illustrated in FIG. 6A). One or moresemantic atoms 1000 may even be stored separately and removed from thevirtual assistant system 10 (e.g., on a removable memory device such asa USB thumb drive), and brought back to the virtual assistant system 10at a future date, at which time the semantic atom(s) 1000 would beinstantly understood by the virtual assistant system 10.

When the virtual assistant system 10 receives an input from a human orfrom a system (service 300) with which it is integrating, the virtualassistant system 10 may convert the input into one or more semanticatoms 1000. If the input is received from a user (at the mobile device100), the input is translated into one or more semantic atoms 1000 atthe assistant interface 110. If the input is received from an externalservice 300, the input is translated into one or more semantic atoms1000 at the corresponding plugin 220. Each plugin 220 includesinstructions to translate between semantic atoms 1000 and theproprietary language of the corresponding service 300. Accordingly, forevery service 300 that the virtual assistant system 10 knows how tointeract with (via a plugin 220), the virtual assistant system 10 knowshow to translate from semantic atoms 1000 language to the datastructures (proprietary language) used by that service 300, and viceversa. These plugins 220, used in conjunction with semantic atoms 1000,allow the orchestration of actions across a vast number of disparateservices 300 a-n with ease.

The assistant server 200 may include any type of master/slavearrangement as predetermined by a user. This may be predetermined by theend-user, or someone that is managing the assistant server 200 can alsoset up the arrangement of the assistant server 200 in a predeterminedway. The assistant server 200 may also be connected to any number ofmobile devices 100 a-n, and may therefore facilitate communicationbetween these mobile devices 100 a-n, as will be described in greaterdetail infra.

As described above, the storage 250 can store various languages, icons,gestures, sensor-commands, key-strokes, voice data commands, programs,applications, etc., as well as libraries including any combinations ofthe above items and commands in the semantic language using semanticatoms 1000. These items and commands are associated with any commandsthat can be instructed by a user by inputting such a command at themobile device 100, which then sends the input command to the assistantserver 200 via the assistant interface 110 after being translated intosemantic atoms 1000 (the semantic language) by the assistant interface110. Alternatively these items or commands can be obtained by the mobiledevice 100 via another source (such as, for example, sensors 160, to bedescribed in more detail below with reference to FIGS. 4A, 5A-5B, and 8)and then presented to the assistant server 200 via the assistantinterface 110 after being translated into semantic atoms 1000 by theassistant interface 110.

As pointed out above, the assistant interface 110 first translates aninput command into one or more semantic atoms 1000 that represent theinput command. In an exemplary embodiment of the present generalinventive concept, a single atom 1000 is generated to represent a singleinput command. However, the assistant interface 110 may instead generatemultiple semantic atoms 1000, for example if the command is being sentto multiple assistant servers 200 a-n or multiple services 300 a-n. Theatom 1000 generated by the assistant interface 110 may itself alsoinclude other atoms 1000, for example if the user is forwarding a song(represented by an atom 1000) to assistant server 200 with instructionsto play the song at a corresponding service 300.

Once the semantic atom(s) 1000 corresponding to the command is/areprovided to the assistant server 200, the atom(s) 1000 may be convertedto the appropriate proprietary language at the corresponding plugin 220.The proprietary language command provided by the plugin 220 containswithin it specific credentials with respect to the service 300, such as,for example, where the service 300 is located, the IP address of theservice 300, and/or any other information that may be required to permitthe virtual assistant system 10 to directly control the desired service300. Thus, each plugin 220 can be created by an engineer to containspecific credentials of the corresponding service 300 required in orderto communicate with and operate the service 300.

In other words, any type of gesture, icon touch, voice, keyboard input,or other input command input at the mobile device 100 or through themobile device 100 (a raw input command) by another sensor type device,can be first translated into one or more semantic atoms 1000 via theassistant interface 110. Once the semantic atom(s) 1000 corresponding tothe input command are recognized within the assistant server 200, theycan be translated at each plugin 220 created for the correspondingexternal service 300 to the proprietary language (including anynecessary credentials) used by the corresponding external service 300.Using the proprietary language format (including any necessarycredentials) of the input command, the intended function/operationdesired to be performed at the corresponding external service 300 can beperformed.

As described herein, semantic atoms 1000 may be converted to proprietarylanguage at the plugins 220. When a semantic atom 1000 is “converted” or“translated” at a plugin 220, one or more proprietary language commandsis/are generated at the plugin 220, and the original semantic atom 1000may be discarded or maintained. Plugins 220 may discard or maintainatoms 1000 depending on the immediate situation, e.g., the content ofthe atom 1000, the type of function(s) to be performed at the service300, etc. Rules on managing atoms 1000, including whether to discard ormaintain atoms 1000, are kept at the plugins 220.

An atom 1000 may be discarded when it is determined that the originalatom 1000 is no longer necessary. For example, if the atom 1000represents a command to turn on a light (a service 300), there no needto keep the original atom 1000 after the corresponding plugin 220 hassent a proprietary language command to the light to turn on. If theoriginal semantic atom 1000 is discarded, the plugin 220 may stillgenerate new semantic atoms 1000, for example based on output datareceived from the service 300. In the example of turning on a light,above, if the plugin 220 discards the original atom 1000, the plugin 220may generate a new atom 1000 representing, e.g., the color of the lightafter the light is turned on.

Alternatively, an atom 1000 may be maintained so that furtherinformation may be added to it. For example, if an atom 1000representing a person is sent to a Facebook™ plugin to look up thatperson in Facebook™, the results of the search (e.g., the person'sFacebook™ profile) may be converted to the semantic language at theplugin 220 and added to the original semantic atom 1000 representing theperson. Adding information to the atom in this way may be referred toherein as “enriching” the atom.

As illustrated, for example, in FIG. 1A, the assistant server 200 maycommunicate directly with the external services 300 a-n by convertingthe received commands expressed in the form of semantic atoms 1000 tothe corresponding proprietary language at the plugins 220. In thisexemplary embodiment, the mobile device 100 is configured to be part ofthe virtual assistant system 10 and communicates with the assistantserver 200 through the assistant interface 110. The assistant server 200can control (via plugins 220 a-n) any number of the external services300 a-n included within the virtual assistant system 10 via an Internetor any other type of wireless connection, as described above, which theassistant server 200 shares with the external services 300 a-n.Furthermore, the assistant server 200 can provide interaction (shareinformation) between each of any number of the external services 300 a-nincluded within the virtual assistant system 10 using the semantic atoms1000 as a common language. These external services 300 a-n may beconnected to the assistant server 200 via an Internet or other type ofwireless connection, as described above.

Alternatively, the assistant interface 110 may directly performfunctions at the services 300 a-n which are associated with a respectiveinput command provided at the mobile device 100. FIG. 1B is a blockdiagram illustrating this exemplary embodiment of the present generalinventive concept. As illustrated in FIG. 1B, the mobile device 100 isconfigured to be part of the virtual assistant system 10 andcommunicates with the assistant server 200 through the assistantinterface 110, similar to the exemplary embodiment of FIG. 1A. However,in this exemplary embodiment, when it is determined that the mobiledevice 100 is within a certain range of a particular external service300 a-n desired to be controlled, or within a common network of theparticular service 300 a-n, the assistant server 200 can communicateback to the mobile device 100 to permit the mobile device 100 to performdirect control of the particular external services 300 a-n, similar tothe way a remote control device would perform these controls (i.e. atelevision and corresponding remote control device).

More specifically, if a location based communication service (forexample, NFC, Bluetooth, etc.) is established between the mobile device100 and a desired external service 300 to be controlled, this“established proximity” communication information can be provided to theassistant server 200 through the assistant interface 110. The assistantserver 200 can then provide control access of the desired externalservice 300 to the mobile device 100 directly. In this situation, theplugin 220 corresponding to the particular service 300 can first embedthe necessary proprietary language information into one or more semanticatoms 1000 so that the assistant server 200 can provide this informationto the mobile device 100 through the assistant interface 110. When theone or more semantic atoms 1000 containing the proprietary languageinformation are received at the assistant interface 110, the assistantinterface 110 may extract the proprietary language information from theatom(s) 1000, which the mobile device 100 can use to control the service300 b. As a result, the mobile device 100 can directly control thedesired external service 300 in the same fashion as a hand-held remotecontrol specifically designed for the service 300 would control theservice 300.

A more detailed overview of the control process of the virtual assistantsystem 10 is provided below with reference to FIGS. 7A and 7B.

It is to be noted that an infinite number of external services 300 a-nmay be accessed and controlled by the virtual assistant system 10, andany number of mobile devices 100 a-n can be included within the virtualassistant system 10 to access and control the infinite number ofexternal services 300 a-n. Access to the services 300 a-n may also beshared between mobile devices 100 (see dotted lines “sharing access” inFIG. 1A), as will be described in more detail infra.

As pointed out above, the semantic language (in the form of the semanticatoms 1000) created within the assistant server 200 acts as a universallanguage to allow communication and exchange of information betweenexternal services 300 a-n through the assistant server 200 viarespective plugins 220 a-n for each service 300 a-n, between the mobiledevice 100 and the assistant server 200, between the mobile device 100and any of the infinite number of external services 300 a-n, and betweenmobile devices 100 a-n. An example of a list of command inputs caninclude, but is not limited to, the following: a voice command can becaptured by a microphone connected to or integrated as part of a mobiledevice 100 (such as, for example, a smartphone or tablet) or othercomputing hardware. Gestures can be captured by, e.g., a camera, motiondetector, proximity detector, infrared sensor, temperature sensor,global positioning device, or any other device that is capable ofcapturing information about a user's movements, which is in turnconnected to a mobile or other computing device. Gestures may also becaptured by wearable, held, or other devices that detect movement byusing accelerometer hardware or other movement-tracking hardware, oreven eye-tracking, lip-tracking or facial-tracking systems. Icons can betouched or clicked either by hand or via a stylus on a touch sensitivedevice (such as that of a mobile device or other touchscreen) or bymeans of a click by positioning a mouse or roller-ball on top of theicon. In addition, it is possible that the user may use voice alone,gestures alone or icons alone, or any combinations thereof to make theassistant perform these tasks. For example, a user may gesture upwardsat a camera to which the assistant has access to indicate that athermostat controlled by the assistant should raise the temperature.Alternatively, for example, this gesture can be detected by means of anarmband or other wearable device.

In creating a command set for a plugin 220, an engineer/developer mayselect from command inputs that correspond to predefined command inputsas understood by the virtual assistant system 10. For example, if oneclosed first and two closed fists are part of the predefined semanticlanguage stored in the virtual assistant system 10, anengineer/developer may select a gesture of closing a first to representthe locking of a door and closing two fists to represent engaging adeadbolt.

As another example, a user may hold a picture up to a camera (associatedwith the user's mobile device 100) and with a waving gesture instructthe assistant server 200 to identify a person in the picture via aface-recognition program (a service 300, for example, an “app”) to whicha plugin 220 has been created. This specific gesture of waving a picturetogether with using such a face-recognition program will have beenpreviously provided within the semantic language of gestures stored inthe storage 250. Then this gesture may be selected by an engineer whilecreating a plugin 220 for a face recognition type service 300.

Because the library of commands programmed at the assistant server 200in the semantic language is constant across a limitless (thousands oreven millions) number of plugins 220, similar commands may emerge forsimilar concepts. As an example, two separate plugins 220 could berespectively created for a door and a billboard, respectively, in apublic square, and stored within the assistant server 200. If, forexample, closing a first is associated with the concept of shutting downor closing up for the day, a user may then utilize a same gesture ofclosing a first to signify locking the door as well as shutting down thebillboard, depending on whether the user wishes to control the door orthe billboard.

Since plugins 220 are created for both the door and the billboard (aswell as an infinite number of other devices, services, programs,applications, etc.), both the door and the billboard are able tounderstand the commands initiated by the user through the mobile device100, as the assistant server 200 acts as a translator or an interpreter(via the semantic atoms 1000, based on the library of gestures, icons,and other command inputs stored in the assistant server 200) between themobile device 100 and the door and the billboard. More specifically, theassistant server 200 acts as the translator between the infinite numberof external services 300 a-n, etc., and the mobile device 100 by usingthe semantic language in the form of semantic atoms 1000. As pointed outabove, the semantic language (the common vocabulary between all devices,services, etc.) is specifically created within the assistant server 200to allow communication between the assistant server platform 210 (seeFIG. 1A) and each of the plugins 220 a-n for all of the respectiveservices 300 a-n in which a plugin 220 is created at the assistantserver 200, and between the assistant server platform 210 and theassistant interface 110.

As an example of the functionality of semantic atoms 1000, when thevirtual assistant system 10 searches a social network (a service 300 a)for people and receives a list of people in response to the search, thevirtual assistant system 10 (specifically, the plugin 220 acorresponding to the service 300 a) may convert the received list ofpeople into semantic atoms 1000 that represent those people. Thesesemantic atoms 1000 can be shown to the user (via the assistantinterface 110), and/or the atoms 1000 can be provided to one or moreother services 300 b-n. Alternatively, or in addition, the atoms 1000can be transmitted to other users of the assistant server 200, or toother users using one or more separate assistant servers 200 a-n incommunication with the assistant server 200. An example of this lattersituation is where there are multiple instances of the assistant server200, as noted above and described in greater detail infra with referenceto FIGS. 6B and 6C.

As another example, a user can be following an artist on Facebook™. Theuser may receive a semantic atom 1000 from Facebook™ representing theartist (translated from the Facebook™ proprietary language by thecorresponding plugin 220). The user can then take the semantic atom 1000representing the artist (in a single command, issued by, e.g., touch,voice, gesture or another command input) and then request from Spotify™,Rdio™ and Youtube™ which songs by that artist are available. Whathappens in this example is that, having retrieved the semantic atom 1000representing the artist, the user performs a command input (icon touch,gesture, etc.) for a “search” command. This command input could be thesame command input for all music/content services used by the virtualassistant system 10. The semantic atom 1000 representing the artist isthen converted into suitable formats, i.e., the correspondingproprietary languages of the selected music/content services 300 a-n(Spotify™, Rdio™ and Youtube™ in this example) by the correspondingplugins 220 a-n, and is provided to the selected music/content services300 a-n along with the “search” command, which is also converted intothe corresponding proprietary languages by the corresponding plugins 220a-n. Searches for the artist may then be made in each of the selectedmusic/content services. In this example, the user is only required tofind the artist in question on Facebook™ and perform a single commandinput to conduct multiple searches in multiple services 300 a-n. Afterthe search is executed, the results may be returned by all of theservices 300 a-n used to the assistant server 200 (via the plugins 220a-n) as one or more semantic atoms 1000 representing the one or moresongs by that artist which were found. These semantic atoms 1000 canthen be sent to other services 300 a-n. For example, the atoms 1000representing the songs may be sent to a music system (such as Sonos™) tobe played. In other words, a service 300 a (Facebook™, in this example)may communicate data to other services 300 b-n (Spotify™, Rdio™ andYoutube™), which may in turn communicate with other services 300 b-n,each service 300 a-n communicating in their respective proprietarylanguages, managed by the assistant server 200 and the correspondingplugins 220.

In the above example of sending a semantic atom 1000 to multiple plugins220, the atom 1000 may be copied and sent simultaneously to severalplugins 220 (Spotify™, Rdio™ and Youtube™ in the above example).Alternatively, the atom 1000 may be sent to a single unified plugin,referred to herein as a “recipe plugin” or “super-plugin,” which isconfigured specifically to interact with several other related plugins220. In the above example the atom 1000 received from Facebook™ could besent to such a super-plugin configured to communicate with music lookupservices (Spotify™, Rdio™ and Youtube™ in the above example). Thesuper-plugin would then handle copying the atom 1000 and sending thecopies of the atom 1000 to the various plugins 220 corresponding to thedesired services. The super-plugin would also receive atoms 1000 backfrom these various plugins 220, and may manage these atoms 1000according to the user's commands and rules stored within thesuper-plugin. For example, the super-plugin may present the atoms 1000to the user, aggregate the atoms 1000 into a single atom 1000, or enrichthe original atom 1000 (representing the information received fromFacebook™) by adding the atoms 1000 received from the other plugins 220to it. Using a super-plugin in this manner may make controlling manyservices 300 simpler for a user: the user only perceives interactingwith one application (the super-plugin), while in fact the user'scommands are being used to control any number of related services 300through the super-plugin.

Sending atoms 1000 to services 300 a-n (via the corresponding plugins220 a-n) can also be described herein as “flinging” or “tossing” atoms1000. Sending atoms 1000 to services 300 a-n may involve a command inputgesture of “flinging” or “tossing,” in which the user gestures towardsan intended service 300 a (e.g., a music player), to command the virtualassistant system 10 to send one or more atoms 1000 to the correspondingplugin 220 a to generate one or more proprietary language commands tosend to that service 300 a. The command inputs may accordingly beperformed more intuitively than conventional methods of manipulatingbuttons. For example, a user may simply gesture towards a music playerwith regards to a selected atom 1000 (corresponding to, e.g., one ormore songs), and the virtual assistant system 10 may understand thecommand input as meaning “play the song(s) represented in this semanticatom 1000 on the music player the gesture is directed towards.” Theseintuitive command inputs may be pre-programmed by plugin developers, ormay be developed by the virtual assistant system 10 itself via adaptivelearning, described in detail infra.

Semantic atoms 1000 may also be put into a collection. For example, agroup of friends can collaborate to decide which movie to see and whichrestaurant to eat at as a group in a given evening. The friends,corresponding to a group of users of the virtual assistant system 10,can then pass a collection of movies and restaurants between themselves.This collection may be represented by one or more semantic atoms 1000which are passed between the users. The collection may itself be asingle semantic atom 1000 which includes other semantic atoms 1000representing individual items (e.g., restaurants and movies). Eachindividual user may have the authority to remove items from thecollection and to add items to the collection. As part of building andmodifying this collection, each user can use whatever underlyingservices 300 a-n they prefer. For example, one user might use IMDB™ toselect movies and another user might use Rotten Tomatoes™ for the samepurpose. In this example, the users eventually agree upon a set ofmovies. At this point a movie can simply be picked from the agreed-uponlist and tickets for that movie may be purchased rapidly through a movieticket service, for example Fandango™. In this example, upon receiving a“purchase” command input from a user, the assistant server 200 mayseparate a single atom 1000 (corresponding to the selected movie) froman atom 1000 representing the list of agreed-upon movies, and send theatom 1000 of the selected movie to a selected movie ticket service witha command to buy one or more tickets corresponding to that movie. Theplugin 220 for the movie ticket service knows how to read and use thesemantic atom 1000 corresponding to the selected movie. Morespecifically, the plugin 220 for the movie ticket service includesinstructions instructing the assistant server 200 how to convert theatom 1000 representing the selected movie into the proprietary languageof the ticket service, and to purchase the corresponding ticketsaccording to the “purchase” command.

Similarly, atoms 1000 may be copied. For example, a single semantic atom1000 representing a command may be copied so that semantic atoms 1000a-n representing the command may be simultaneously presented to aplurality of plugins 220 a-n, to carry out the command at a plurality ofservices 300 a-n. Copying atoms 1000 in this manner may be performed bythe assistant server 200, or by, e.g., a super-plugin communicating withseveral other plugins 220, as described above. In the example above ofdeciding on movies, a single user may buy a ticket to the agreed-uponmovie for themselves, and may send copies of one or more atoms 1000involved in the transaction to other users to similarly purchasetickets. The atom(s) 1000 copied in this manner may represent, forexample, the movie, the act of purchasing a ticket (for example,specifying a particular ticket service). The act of copying atransaction in this way may be referred to as a “mimicking transaction,”and allows multiple users to perform the same action, e.g., purchasing aticket to a particular movie. A mimicking transaction allows differentusers to use different payment credentials. Alternatively, the paymentcredentials used for one transaction may also be mimicked (representedas semantic atoms 1000, copies of which are sent to other users).Payment credentials may mimicked if, for example, the first user agreesto pay for the others users' tickets.

Users can share atoms 1000 with each other in a messaging or “chat”format, where the users are in constant conversation with each other.The users can obtain atoms 1000 generated by various services 300 a-nand share the obtained atoms 1000 with other users via messaging. Oneuser may send, or “message,” one or more atoms 1000 to one or more otherusers, who may then select these one or more atoms 1000 and use the oneor more atoms 1000 with one or more services 300 a-n. For example, afirst user can send a second user an atom 1000 corresponding to a movie.The second user can use the received atom 1000 to find theaters whichare nearby and playing the movie, for example by sending the receivedatom 1000 to a search service. The second user may receive the resultsof this search, represented as one or more additional atoms 1000including, e.g., movie times and locations. Then the second user canshare these one or more received atoms 1000 with the first user as partof the chat session. Alternatively the second user may message the oneor more received atoms 1000 to one or more other users, either as a newchat that does not include the first user, or as part of the originalchat which includes the first user. Groups can also be defined, whichinclude a set of users.

Any user in a defined group can message or “toss” atoms 1000 into acommon chat between the users in the group. Any atom 1000 can be takenfrom one chat between two or more users and “tossed” into a chat betweendifferent users by a user who is common to both chats. This situation isillustrated in FIG. 2A. FIG. 2A illustrates 5 users, with correspondingmobile devices 100 a-e. It will be understood that any number of usersand mobile devices 100 a-n may be included without departing from thepresent general inventive concept. In FIG. 2A, there are two chats: onebetween the users of mobile devices 100 a, 100 b, and 100 c, and anotherbetween the users of mobile devices 100 a, 100 d, and 100 e. Since theuser of mobile device 100 a is a party to both chats, that user may“toss” semantic atoms 1000 from, e.g., the chat between mobile devices100 a, 100 d, and 100 e into the chat between mobile devices 100 a, 100b, and 100 c.

As illustrated in FIG. 2B, services 300 a-n can also “participate” inchats through the respective plugins 220 a-n, receiving and presentingatoms 1000 to the chat in a manner similar to a user. As illustrated inFIG. 2B, a dropcam plugin 220 a, corresponding to dropcam 300 a, isparticipating in a chat with several users (corresponding to mobiledevices 100 a-c). It will be understood that any number of plugins 220a-n, corresponding services 300 a-n, and mobile devices 100 a-n may beincluded in the chat in this manner without departing from the presentgeneral inventive concept. Furthermore, although the plugin 220 a isillustrated as stored in the assistant server 200 in FIG. 2B, the plugin220 a may be stored elsewhere, for example, on another mobile device 100which participates in the chat. The dropcam plugin 220 a in FIG. 2B maypresent, e.g., a picture atom 1000 in that chat, representing a picturetaken by its associated dropcam 300 a. All users in the chat wouldreceive the atom 1000 from the dropcam 300 a, and any user in the chatcan then take that atom 1000 and send it to other users, including usersin another chat not including the plugin 220 a.

As another example, commercial services can monitor deals available instores in a geographic area and toss atoms 1000 representing those dealsto a user or into a chat of several users, potentially based on thegeographic location of those users. If the atoms 1000 arehuman-readable, e.g., they may include a title indicating they are adeal from a retailer, users can understand the meaning of the receivedatoms 1000 without needing to send the atoms 1000 to one or moreservices 300 a-n first. Users in the chat can also discuss the receiveddeals within the same chat, and message the atoms 1000 representing thedeals to each other or to other users in different chats.

When services 300 “participate” in chats, those services 300 may alsoreceive one or more atoms 1000 from users. In the above example of usersagreeing upon a list of movies and buying tickets for one agreed-uponmovie, the movie ticket service could be participating in the chatbetween the users, in which case any user in the chat could send theatom 1000 representing the selected movie to the ticket service 300 inthe context of the chat. This action of sending the movie atom 1000 tothe ticket service 300 would be interpreted by the assistant server 200as a command to buy tickets for the movie represented by the atom 1000,and a corresponding command would be sent to the ticket service 300through the ticket service's plugin 220. In the example of the dropcam300 a illustrated in FIG. 2B, users in the chat may send commands to thedropcam plugin 220 a through the chat, and these commands may beconverted to the dropcam's proprietary language and carried out at thedropcam 300 a.

One or more users in a chat session may also invite a plugin 220 toparticipate in a chat session, invisibly from the other users in thechat session, based on the users' authority of access to those plugins220 a-n. Basically, the users may bring one or more plugins 220 a-n intothe chat session and make use of these plugins 220 a-n, withoutextending authority to other participants of the chat session or evenallowing other participants in chat session to be aware of the plugins'participation. Plugins 220 a-n in this case may include the users'individual assistant interfaces 110 for the virtual assistant system 10.Users may also share their plugins 220 a-n with the other participantsin the chat session as needed. This sharing of plugins 220 a-n withother users through a chat session may be selective, i.e., plugins 220a-n may be shared with all the users in a messaging session or only asubset of the users in a messaging session, as desired by the userowning the plugins 220 a-n.

Users can also collaborate to assemble data through messaging. As anexample, one user can obtain one or more semantic atoms 1000representing an order for a pizza and message the atom(s) 1000 toanother user, who then adds toppings to the pizza, in effect addingadditional information to the atom(s) 1000 representing the pizza. Theatom(s) 1000 may be messaged between multiple users in this way, eachuser modifying the atom(s) 1000 (e.g. by adding or removing toppings)until the users are ready to order. At that point, one of the users maysend the atom(s) 1000 representing the pizza order to a service 300, forexample Pizza Hut™, to carry out the corresponding operation, in thiscase ordering the pizza. As part of being messaged to Pizza Hut™, thesemantic atom(s) 1000 representing the pizza are sent to the assistantserver 200, which in turn sends the data to the Pizza Hut™ plugin 220 togenerate a corresponding command in the Pizza Hut™ proprietary language,which is thereafter sent to Pizza Hut™ to order the pizza. Notably,these operations can occur automatically without direct userinteraction. In the users' perception, they messaged between each otherregarding what kind of pizza to buy, and then messaged their agreed-uponpizza order to Pizza Hut™.

Similarly, a retailer might message a user an offer, for example anoffer to buy a product at a discount. That offer would be represented asone or more semantic atoms 1000 sent into a chat by a service 300 (theretailer in this example). A user may be able to message the atom(s)1000 representing the offer to another user, and this other user mayredeem the offer by sending the atom(s) to the retailer's plugin 220with a command to redeem. This redemption may be performed according toa relevant context. Whether the offer has been received from anotheruser may be a context which affects the offer, such that the offerchanges as it is messaged from one user to another. For example, if theinitial offer from the retailer to the first user was to buy a softdrink at 20% off, the same offer, after being messaged to another user,may allow the other user to buy a soft drink at 50% off. Alternatively,the offer, after being messaged to the other user, may allow the otheruser to buy coffee at 50% off from a coffee outlet owned by the sameretailer. In other words, the number of times data has been messaged mayalso be included in the semantic atom(s) to be used as a context by thecorresponding plugin 220 to interpret as a command. The transmission ofatom(s) 1000 (corresponding to the offer) may also be trackable. A userthat shares an offer with many of their friends and then redeems theoffer might be rewarded with another offer, for example for a freeproduct, based on their friends' sharing and use behavior.

Tracking of the number of times atoms 1000 representing an offer havebeen messaged between users can also be used to gamify commerce quiteextensively. For instance, a user that is successful in referring manyfriends to a new service might gain an elite status with that servicebased on the sharing and use behavior of that user's friends. In all ofthese cases, the same data (the offer) is being translated differentlyin different contexts, and potentially between different plugins 220a-n.

This advertising could be more open as well. For example, a retailermight announce that 50 units of an item are available for free. Then oneor more users can request and receive one or more atoms 1000corresponding to the offer from the retailer's plugin 220 (which mightbe accessible via chat also) if the offer is still available (in thisexample, if any of the 50 units of the item are still available). Theone or more users may be able to then transfer these one or more atoms1000 (corresponding to the offer along with the authority to redeem theoffer as described above) to other users. Receiving and redeeming theoffers might require the receiving users to also install or communicatewith a plugin 220 that belongs to the issuing retailer or,alternatively, such installation and/or communication might not berequired. Messaging according to the present general inventive conceptas described herein then becomes a way by which a retailer can reachmany potential customers and generate “buzz” by targeting highlyinfluential individuals with transferrable offers.

Two advantages of semantic atoms 1000, among many, is that theyeliminate the need to input data over and over again they and enablerapid reuse of data across very different systems by creating a commonlanguage. Semantic atoms 1000 may represent objects and concepts a humanmay understand, such that the atoms 1000 on their own may be meaningfulto a human. As such, semantic atoms 1000 may be almost entirely humancomprehensible when examined. For example, a semantic atom 1000corresponding to a movie may contain information about the movie'stitle, synopsis, actors, director, language, ratings, genre, runningtime, release date and available formats thereof. This information canbe rendered directly to a user. That is, the information about the item(the movie in this case) may be in a format the user can understand,such as, for example, text. In the same way, semantic collections(corresponding to collections of semantic atoms 1000) may also be humanviewable. Despite the foregoing, however, there may be other informationin a semantic atom 1000 that is not human-comprehensible, even thoughthe semantic atom 1000 may still carry meaning. For example, a semanticatom 1000 corresponding to a person may contain eigenvalues thatmathematically represent that person's face in order to make that personrecognizable to a camera. In this example, a human looking at thoseeigenvalues would not be able to visualize the person directly, althoughthe eigenvalues still contain meaningful information, in this examplethe information that allows a camera to visualize that person.Furthermore, not all the information containable in a semantic atom 1000is necessarily contained in it. Sometimes only a subset of the availabledata is included in the atom 1000. For example, a movie semantic atom1000 might only contain the name but not the synopsis of the movie.

Semantic atoms 1000 can also be combined or merged. An exemplaryembodiment of this use of atoms 1000 is illustrated in FIG. 3A. Asillustrated therein, when a service 300 a, such as, for example,Facebook™, is accessed, the service 300 a may generate (through itscorresponding plugin 220 a) a semantic atom 1000 a 1 corresponding to,e.g., a person, the semantic atom 1000 a 1 including information aboutthat person (e.g., name, picture, birthday, etc.). When a differentservice 300 b, for example, an e-mail service, is accessed, this newservice 300 b may generate (through its corresponding plugin 220 b) asemantic atom 1000 a 2 corresponding to the same person, but withdifferent information (e.g., e-mail address, telephone number, etc.).The virtual assistant system 10 (for example, the assistant server 200)may merge these two atoms 1000 a 1 and 1000 a 2, corresponding to thesame person but including different information about that person, intoa single atom 1000 a 3 representing that person. This combination ofatoms 1000 may be performed based on rules maintained in the assistantserver 200 (e.g., on one or more plugins 220 stored at the assistantserver 200). Any number of semantic atoms 1000 a-n may be combined inthis manner. Additionally, although FIG. 3A only illustrates twoexternal services 300 a and 300 b for convenience, it will be understoodthat any number of external services 300 a-n, and corresponding plugins220 a-n, may be used in this manner to combine semantic atoms 1000.Furthermore, although FIG. 3A illustrates the plugins 220 a-n as storedin the assistant server 200, and that the assistant server 200 combinesthe atoms 1000 a-n (atoms 1000 a 1 and 1000 a 2 as illustrated in FIG.3A), the plugins 220 a-n may alternatively be stored at the mobiledevice 100, which may also combine the atoms 1000 a-n. Furthermore, ifthere are multiple instances of the assistant server 200, differentindependent instances 200 a-n may collaboratively merge semantic atoms1000 a-n to create a semantic atom 1000 that is richer with informationor better suited to a particular task.

Semantic atoms 1000 may also be nested according to rules maintained inthe plugins 220. In the exemplary embodiment of the present generalinventive concept as illustrated in FIG. 3B, semantic atom 1000 arepresents, for example, a music playlist. Each one of the songs in thisplaylist could itself be an atom 1000 b-n that is contained in thesemantic atom 1000 a representing the playlist. In FIG. 3B, three nestedatoms 1000 b, 1000 c, and 1000 d are illustrated for convenience, but itwill be understood that any number of atoms 1000 b-n may be nested inthis manner without departing from the present general inventiveconcept. Furthermore, each nested atom 1000 may further include othernested atoms 1000. As illustrated in FIG. 3B, for example, atoms 1000 eand 1000 f may be nested within atom 1000 b. If atom 1000 b correspondsto a song, for example, atoms 1000 e and 1000 f may correspond to, e.g.,lyrics and price of that song.

“Larger” atoms 1000, generated by combining or nesting atoms 1000 a-n,can be exchanged between people and services 300 also, thus greatlysimplifying chats in which one user may wish to message several items atonce to another user. For example, user A can send user B the playlistatom 1000 a illustrated in FIG. 3B. User B may pick one of the songatoms nested inside playlist atom 1000 a (for example, song atom 1000 b)and send the selected song atom to user C, who could then in turn addthe song atom to a different playlist atom.

Semantic atoms 1000 may also represent actions. For example, a user maysend another user a location request. This location request may containa variety of information, including time needed, rationale for need andaction to be taken upon receipt. Another atom 1000 could represent theact of leaving home. This “leaving home” atom 1000 may be sent toanother user for example, enabling the other user to use the atom 1000to take a set of consequent actions. For example, if a renting guestdeparts from a rental property and sends a “departing event” semanticatom 1000 to the owner, the owner can then send this “departing event”atom 1000 to the lights, heating, security, WiFi and other services 300a-n to automatically take actions in the rental property associated withsuch an event. In this example, sending the “departing event” atom 1000to the various services 300 a-n may cause the lights to turn off, turnthe heating to a lower temperature setting, and so on.

Semantic atoms 1000 can also be saved for various purposes. For example,various WiFi atoms 1000 that contain the passwords to access variousWiFi access points might be persisted by a user for future use. Asdescribed above, atoms 1000 may be stored in the mobile device 100 orthe assistant server 200, as well as, e.g., a removable storage mediumor other storage medium.

Semantic atoms 1000 can also be discarded after viewing. For example, auser might run a search on LinkedIn™ and see a number of people as aconsequence which are represented by semantic atoms 1000 in the virtualassistant system 10, but not do anything further with these atoms 1000.In this case, the unused atoms 1000 are simply discarded by theassistant server 200, plugin(s) 220, and service(s) 300.

FIG. 4A is a block diagram illustrating an exemplary embodiment of amobile device 100 in detail for purposes of describing how the mobiledevice 100 is configured to be a part of the virtual assistant system 10illustrated in FIGS. 1A and 1B.

The mobile device 100 may include the assistant interface 110, acommunicator 105, a controller 120, a central processing unit (CPU) 130,a storage unit 140, a user interface 150, a sensor 160, at least onespeaker 170, at least one microphone 180, and at least one camera 190.

Referring to FIG. 4A, the communicator 105 can be used to establish aconnection with the assistant server 200 and/or an external service 300.The communicator 105 may include an antenna or any type of communicationdevice to establish the connection with the assistant server 200 and theexternal service 300 using Wi-Fi, Wi-Fi direct, NFC, Bluetooth, RFID,Ethernet, FireWire, universal serial bus (USB), high definitionmultimedia interface (HDMI), or any other type of wireless or wiredcommunication method, but is not limited thereto. The communicationsbetween the mobile device 100, the assistant server 200, and anyexternal service(s) 300 may be performed via the Internet, includingcloud computing applications, satellite, etc.

The controller 120 may control communications between the mobile device100 and the assistant server 200, as well as control communicationbetween the mobile device 100 and the external services 300 a-n undercertain conditions being met, such as a location based communicationservice (e.g., near field communication (NFC), Bluetooth, etc.) beingestablished, as described above with respect to FIG. 1B.

The controller 120 determines which service 300 the assistant interface110 is communicating with in order to interpret a received requestwhether the request is received in a proprietary language format or assemantic atoms 1000. Although the service which the assistant interface110 is communicating with is often the assistant server 200, thelibraries of possible inputs (icons, gestures, voice commands, withinterpretations) could be obtained from a different server or fromanother assistant interface 110 a-n on a separate mobile device 100 a-n.For example, multiple assistant interfaces 110 a-n could workcollaboratively: i.e., once one assistant interface 110 a has learnedhow to interpret an input, this assistant interface 110 a could sharethe input(s) with another assistant interface. Accordingly, thecontroller 120 may determine what device the assistant interface 110 iscommunicating with in order to provide these features.

The controller 120 can thus make a determination that the assistantinterface 110 should reach out to an external source other than theassistant server 200 under certain circumstances. For example, if theassistant interface 110 is determined by the controller 120 to bereceiving an input request that requires information from anotherassistant server 200 a-n or another assistant interface 110 a-n, thecontroller 120 may dictate to the assistant interface 110 to reach outto that external other source. The assistant interface 110 wouldaccordingly connect to the external other source by way of thecommunicator 105 and present the request. The external other source willcheck the full set of libraries to interpret this request (gesture, icontouch, voice data, etc.), and if found, the external other source willrespond to the assistant interface 110 and tell it what to do withsemantic atoms 1000 (with proprietary language embedded therein). Atthis point if the mobile device 100 is operating through the assistantserver 200 as illustrated in FIG. 1A, the assistant interface 110 willprovide the command to the assistant server 200 to perform the commandon the intended external service 300. Alternatively, if the mobiledevice 100 is operating in a local control mode as illustrated in FIG.1B, the assistant interface 110 can then connect with the service 300through the communicator 105 and directly control the intended externalservice 300 using the properly obtained proprietary language.

The CPU 130 may request the virtual assistant system 10 to runapplications and/or programs, perform various functions of the mobiledevice 100, and adaptively learn various commands based on the user'spreferences, among various other uses. The CPU 130 may also run variousoperating systems that may be changed, altered, or personalized, basedon the user's preferences. Further, when there are a plurality ofexternal services 300 a-n in which the user desires to control, the CPU130 may perform various functions specifically corresponding to theplurality of external services 300 a-n, based on respective usercommands.

The storage unit 140 may store the commands learned by the CPU 130, aswell as the various languages, icons, gestures, sensor-commands,programs, applications, commands, and libraries including anycombinations of the above items, that are downloadable by the mobiledevice 100 from the assistant server 200. For example, if the mobiledevice 100 does not possess a specific command among the library ofcommands within the assistant server 200 to control a particular service300, the mobile device 100 can download any single command or set ofcommands from the assistant server 200, at which point the mobile device100 can directly control the service 300, or at least have a moreextensive library of commands to understand one or more commands beinginput to the mobile device 100. In other words, if a user of the mobiledevice 100 makes, e.g., a gesture or provides a voice command which isnot a readily known command to the mobile device 100, the assistantserver 200 can recognize the correct command by checking within theextensive library of commands, while considering various contexts of thecommand and the user information collected via the history ofinformation previously gathered, as described infra. Then, with the helpof a corresponding plugin 220, the assistant server 200 can generate thecommand as one or more semantic atoms 1000 with the proprietary languageembedded therein. The atom(s) 1000 representing the command can then beprovided to the mobile device 100, at which point the mobile device 100can download the correct command into the storage unit 140, translatethe command to the proprietary language of the service 300 via theassistant interface 110, and then perform an intended function at theservice 300 a through the communicator 105 by using the proprietarylanguage of that service 300.

The user interface 150 allows a user to perform various functions on themobile device 100, including displaying selectable icons, makingtelephone calls, accessing the Internet to view various web pages,playing multimedia such as music and movies, viewing pictures, runningprograms, controlling one or more external services 300 a-n, accessingthe assistant server 200, modifying the mobile device 100, etc., but isnot limited thereto. The user interface 150 may include a screen, akeyboard, a keypad, a mouse, a trackball, a touch screen, an audiointerface (for example with the at least one microphone 180), a holograminterface, a sensing interface, etc., but is not limited thereto.Moreover, the user interface 150 may include any combinations of thevarious types of aforementioned user interfaces.

The sensor 160 may be configured to sense one or more inputs, forexample a sound, gesture, or touch of the user. The sensor 160 may workin conjunction with facial recognition software running on the CPU 130to allow the mobile device 100 to recognize facial expressions of theuser, or a ring type device, such as, for example, Fin™, which is aBluetooth enabled device that can provide commands from gestures made bya user. In this example, the commands can be received by the CPU 130 andtranslated into one or more sematic atoms 1000 by the assistantinterface 110. Additionally, the sensor 160 may include other types ofsensing devices that sense movement of the mobile device 100 itself,such as, for example, a gyration, pressure, or proximity of the mobiledevice 100 to a service 300 to be controlled. Moreover, the sensor 160may include global positioning system (GPS) technology to sense alocation of the mobile device 100. The sensor 160 may include a singlesensor having multiple functions, or may be divided into a plurality ofdifferent sensors based on the user's preference. The sensor 160 isdescribed in greater detail below with reference to FIG. 5A.

The mobile device 100 may further include the at least one speaker 170to allow the user to hear audio emitted from the mobile device 100, theat least one microphone 180 to allow the user to speak into the mobiledevice 100, and the at least one camera 190 to allow the user to captureimages. The at least one speaker 170 and the at least one microphone 180may be used to perform telephone operations, and may also be used toreceive audio commands (for example voice commands) from the user thatmay be processed by the CPU 130. The at least one camera 190 may be usedto photograph still or moving objects, to perform webcam communicationswith other users using applications such as Skype or GChat, or toreceive/read gestures and/or facial features from the user that may beprocessed by the CPU 130.

The assistant interface 110, as described above with respect to FIGS.1A-1B, can perform the operations of a plugin by translating inputcommands into one or more semantic atoms 1000 and can communicate withthe assistant server 200 using the semantic atoms 1000. Furthermore, theassistant interface 110 can translate any of the commands received atthe mobile device 100 by any form of the sensor 160 thereof or receivedby the mobile device 100 by way of an external sensor, such as, forexample only, the wearable ring as described above. The assistantinterface 110 also translates data received from the assistant server200 into a format which may be understood by the mobile device 100(effectively, the proprietary language of the mobile device 100), andwhich may be displayed for example on the user interface 150.

As pointed out above, FIG. 4B illustrates an exemplary embodiment wherethe mobile device 100 of FIG. 4A may include the assistant server 200therein as well as the other devices illustrated in FIG. 4A. In thiscase, the assistant interface 110 is not required to communicate withany backend-type server system in order to understand the actionsrequired to take in order to perform any functions at any services 300a-n when operating in a local mode. Thus, in this exemplary embodimentillustrated in FIG. 4B, the virtual assistant system 10 may be entirelyself-contained in a mobile type device 100, as described above. In theembodiments described above with reference to FIGS. 1A-1B and 2A, theinteraction between the assistant server 200 and the assistant interface110 would normally occur over the web. However, in this exemplaryembodiment illustrated in FIG. 4B, the interaction between the assistantserver 200 and the assistant interface 110 occurs within the mobiledevice 100 itself. Furthermore, the storage 250 associated with theassistant server 200 is also provided on the same hardware as the mobiledevice 100, and is used by the assistant server 200 similar to the waythe assistant server 200 uses the storage 250 as described in FIGS.1A-1B. Thus the storage 250 in FIG. 4B is functionally equivalent to thestorage 250 described in FIGS. 1A-1B. Although many of the components ofthe mobile device 100 of FIG. 4A are not illustrated in the mobiledevice 100 of FIG. 4B, such as the communicator 105, the CPU 130, thecontroller 120, the U.I. 150, the sensor 160, the storage unit 140,etc., in order to highlight the differences between the mobile device ofFIG. 4A and FIG. 4B, these components are also included in the mobiledevice of FIG. 4B.

A virtual assistant system 10 according to this exemplary embodiment ofFIG. 4B is valuable in cases where communication is poor, such as remoteapplications, military applications, etc. Since the mobile device 100and the assistant server 200 of this exemplary embodiment do not rely onwireless communications between each other to perform functions atservices 300 a-n, the probability of failure in controlling an intendedservice 300 as a result of poor wireless communication between themobile device 100 and assistant server 200 can be eliminated.Furthermore, users who do not want their data to be stored at someremote location may also use this type of setup of the virtual assistantsystem 10 for security purposes, or other similar reasons. Since noassistant information is being sent wirelessly, the virtual assistantsystem 10 (and by extension, the user's information) is more secure.This type of setup can also be very valuable in automobiles or aircraftwhich face varying connectivity situations, e.g., sometimes becomingdisconnected from wireless service.

As an alternative to the exemplary embodiment illustrated in FIG. 1B,the “mobile device” 100 may instead be implemented on one or morestationary devices, such as an Amazon Echo™ style device that isconnected to the assistant server 200 or a cloud (which is connected toor serves as the assistant server 200). As such, a user may not requirea handheld device with the assistant interface 110 to connect to theassistant server 200. Instead, stationary devices may be positionedaround various points in an area. These stationary devices can functionas the “mobile device” 100 described above, and may receive commandinputs (e.g., a user's voice, gestures, etc.), interact with the user,and perform functions according to the command inputs. As an example, ifa user is in a building there may be one or more stationary devices,each of which may have their own assistant interface 110, positioned onevery floor. This arrangement of stationary devices allows a user tocommand the virtual assistant system 10 to perform various tasks andcontrol external services 300 a-n, similarly to the exemplary embodimentdescribed above with reference to FIGS. 1A-1B. According to an exemplaryembodiment of the present general inventive concept, the stationarydevices may also communicate with each other.

Stationary devices function similarly to the mobile device 100.Accordingly, it will be understood that the term “mobile device,” asused herein, may also include stationary devices. FIG. 17, discussed indetail infra, illustrates an exemplary embodiment of the present generalinventive concept including multiple devices (e.g., sensors havingseparate assistant interfaces 110 a-n), which may function as stationarydevices as described above.

FIG. 5A is a block diagram illustrating an exemplary embodiment of thesensor 160 of the mobile device 100 illustrated in FIG. 4A. The sensor160 as illustrated in FIG. 5A can be provided in the form of a pluralityof different sensors.

The sensor 160 may include various different sensors located at variouslocations on the mobile device 100, and may include a touch screen 161,a pressure sensor 162, a motion sensor 163, a heat sensor 164, aproximity sensor 165, a gyration sensor 166, and a global positioningservice (GPS) sensor 167.

The touch screen 161 is a display that displays items, icons, and mediathereon, while allowing the mobile device 100 to detect a location ofthe user's finger, stylus, etc., when in contact with the touch screen161. As such, when the user moves their finger around the touch screen161, various commands can be performed on the touch screen 161, such asscrolling through menu items, selecting icons, flipping pages, etc., butis not limited thereto.

The pressure sensor 162 allows the mobile device 100 to detect apressure, for example a pressure at which the touch screen 161 isdepressed. As such, different pressures sensed by the pressure sensor162 can result in different commands executed by the mobile device 100.The pressure sensor 162 may be a component that is separate from thetouch screen 161, or may be integrated therein.

The motion sensor 163 can detect a motion of the user or a motion of themobile device 100 with respect to the user or one or more externalservices 300 a-n. As such, the motion sensor 163 can actively track themovement of the user and/or the mobile device 100. The motion sensor 163may be a component that is separate from the at least one camera 190, ormay be integrated therein.

The heat sensor 164 can sense heat from an outside source, which can beprocessed and interpreted as data by the CPU 130.

The proximity sensor 165 can sense a proximity of the mobile device 100with respect to other objects and/or users. The proximity sensor 165 maybe a component that is separate from the at least one camera 190, or maybe integrated therein.

The gyration sensor 166 can sense a motion and/or a direction of motionof the mobile device 100 itself in a three-dimensional space. Morespecifically, the gyration sensor 166 can sense how the mobile device100 is moving, thereby allowing the mobile device 100 to be used as awand-like apparatus to control the external service 300, for example.For instance, the gyration sensor 166 may sense that the user moves themobile device 100 from a low position to a high position, and then maysend a command through the assistant server 200 to control a television(corresponding to an external service 300) to increase its volume.Alternatively, the motion sensor 163, the proximity sensor 165, and theat least one camera 190 may also be utilized to sense that the mobiledevice 100 has moved from a low position to a high position and send acorresponding command to the assistant server 200.

The GPS sensor 167 can sense a location of the mobile device 100 usingGPS technology, in order to substantiate an exact location of the mobiledevice 100. When the location of the mobile device 100 is substantiated,the assistant server 200 can determine which external services 300 a-ncan and/or should be controllable by the mobile device 100, and whichtypes of commands can and/or should be utilized to perform the controls.

According to an exemplary embodiment, if a user is operating the mobiledevice 100 outside his/her house and wants to open a garage door (aservice 300 a) which is wirelessly connected to the assistant server200, the user may give a “thumbs up” hand sign to the at least onecamera 190, or using another form of sensor, at which point theassistant interface 110 translates and provides the command as asemantic atom 1000 to the assistant server 200 to communicate thecommand to the garage door plugin, causing the garage door to open. The“thumbs up” hand sign is interpreted to signify “garage door open” atthis point because the GPS sensor 167 can sense that the mobile device100 is outside the user's house. However, if the user is inside thehouse, the same “thumbs up” hand sign can be used to close the garagedoor at this point because the GPS sensor 167 can sense that the user isinside the house. Therefore, the same hand sign or other gesture can beused to perform different functions corresponding to a single externalservice 300, based on a location of the mobile device 100 sensed by theGPS sensor 167.

As another exemplary embodiment, a sprinkler system service 300 a couldbe controlled differently based on a location of the mobile device 100.More specifically, if the GPS sensor 167 senses that the mobile device100 is near the user's home and the user touches a sprinkler system icondisplayed on the user interface (UI) 150 of the mobile device 100, thesprinkler system can be shut off to allow the user to walk on the lawnwithout getting wet. If the user is far from the home and the usertouches the same sprinkler system icon displayed on the UI 150 of themobile device 100, the sprinkler system may turn on, causing water tospray onto the lawn. Alternatively, the user could preset an automatedsprinkler system function based on the GPS sensor 167 sensing a locationof the user.

It will be understood that the sensor 160 may include any combination ofthe sensing devices illustrated in FIG. 5A, or additional sensingdevices not illustrated, according to the particular exemplaryembodiment of the present general inventive concept. For example,gestures can be detected by the at least one camera 190, as well as aphoto-detection device, a wearable device such as an armband or Fin™, orusing an accelerometer or other movement detection device in the mobiledevice 100. In the exemplary embodiment of the present general inventiveconcept illustrated in FIG. 5B, the mobile device 100 is connected to aplurality of external input sensors and devices. For example, the mobiledevice 100 may be connected to external sensors in a user's home,office, or car in order to receive data therefrom which it would not beable to receive with its own sensor 160. External sensing devices, suchas a keyboard, camera, GPS, wearable sensor, microphone, and/or infraredsensor would connect the mobile device 100 to alternate sensing devicesto accept a user input. In this way, the functionality of the user'smobile device 100 can be greatly expanded.

Furthermore, the mobile device 100 may receive input commands fromexternal devices, such as one or more displays, speakers, and actuators.The inputs from these external devices can be processed by the CPU 130to determine such a command being received from the respective externaldevice.

As described above, gestures may also be captured by wearable, held, orother devices that detect movement by using accelerometer hardware orother movement tracking hardware, or even eye-tracking, lip-tracking orfacial-tracking systems. Icons are usually touched or clicked either byhand or a stylus on a touch sensitive service (such as that of the userinterface 150 of the mobile device 100 illustrated in FIG. 4A, or someother form of a touchscreen) or by means of a click by positioning amouse or roller-ball on top of the icon. The user may use any commandinput or combination thereof, for example voice alone, gestures alone oricons alone, or any combinations thereof, to make the assistant server200 perform these tasks at the service 300 (e.g., the above example ofthe combination of using a GPS together with a sensor to sense the“thumbs-up gesture”). As another example, a user may gesture upwards ata camera to which the assistant server 200 has access to indicate that athermostat controlled by the assistant server 200 should raise thetemperature. Or this gesture can be detected by means of an armband orother wearable device.

FIG. 6A is a block diagram illustrating an exemplary embodiment of anassistant server 200 in detail for purposes of describing how theassistant server 200 is configured to be a part of the virtual assistantsystem 10 illustrated in FIGS. 1A and 1B.

The assistant server 200 illustrated in FIG. 6A may include theassistant server platform 210 and the plugins 220 a-n.

The assistant server platform 210 may be any device or combinationthereof that allows a software developer to create the plugins 220 atthe assistant server 200. The assistant server platform 210 may, forexample, be a CPU, multiple CPUs, or other computing device used toprocess commands as well as communicate with services 300 a-n viasemantic atoms 1000. Similarly to the CPU 130 in the mobile device 100,the assistant server platform 210 can be continuously running a learningalgorithm or algorithms.

Although plugins 220 a-n are illustrated in FIG. 6A as being separatefrom the assistant server 200, this illustration is provided merely toillustrate the infinite number of plugins 220 a-n that may be createdand their respective associations with specific services 300 a-n. Theseplugins 220 a-n are generally created at the assistant server 200.

As discussed above, the storage 250 may be provided on the same hardwareor cloud service as the assistant server 200, or may be separatelylocated and accessible by the assistant server 200, as illustrated inFIG. 6A. The storage 250 may be, for example, a hard drive included inthe assistant server 200, or an external storage device which isconnected to the assistant server 200, for example a flash drive or acloud storage server. The storage 250 may also be a plurality of storagedevices connected to the assistant server 200. The storage 250 storesthe library of commands 230 used to control the services 300 a-n.

The library 230 may be divided into multiple divisions for differenttypes of inputs. For example, the library 230 may be divided into alibrary of icons, a library of voice/word inputs, and a library ofgestures.

The storage 250 may also include separate command sets 240 a-ncorresponding to services 300 a-n. Each of the numerous command sets 240a-n includes the specific commands (associated with inputs) which may beused to control a particular one of services 300 a-n.

In order to control a particular service 300, the user may connect themobile device 100 to the assistant server 200 and download theappropriate command set 240. This connection may be achieved via theInternet by using a Wi-fi, Wi-fi direct, NFC, Bluetooth, RFID, or anyother type of wireless or wired connection. The downloaded command sets240 may be saved in the storage unit 140, either automatically or aftera user acceptance command. Once downloaded, the command sets 240 may beused by the assistant interface 110 to translate received inputs intoone or more semantic atoms 1000.

The storage 250 may also store usage data on the virtual assistantsystem 10, including context data (such as, for example, a time or auser's physical location when certain commands are entered), whichinputs are given incorrectly (which inputs are immediately undone,etc.), and so on. This usage data may be made available to plugindevelopers so that the developers may then update the library 230accordingly to reflect this usage data.

The storage 250 may also store updates to the library 230 generatedthrough use of the virtual assistant system 10. As noted above anddiscussed in detail infra, the virtual assistant system 10 may defineadditional inputs, such as gestures, depending on user behavior, whichmay be stored in the storage 250.

The assistant server 200 illustrated in FIG. 6A is a centralizedarrangement comprising a single node or server. However, as noted abovethe assistant server 200 may also be a distributed deployment, or“multiple instances” of the assistant server 200, comprising multiplenodes or servers communicating with each other via semantic atoms 1000.

Multiple instances of the assistant server 200 can comprise a network ofhierarchically arranged servers 200 a-n. This network could utilizedifferent instances of the assistant server 200 to handle differentrequests, e.g., requests relating to different things. For example,where one server (corresponding to one instance of the assistant server200) contains plugins 220 relating to a subset of services 300 a-n, forexample physical devices, another instance of the assistant server 200could contain plugins 220 relating to a different subset, for examplesocial media services and applications. A third instance of theassistant server 200 can route requests to and compile responses fromthese underlying servers (the underlying instances of the assistantserver 200). The user perceives interacting with a single entity, i.e.,a single assistant server 200.

When there are multiple instances of the assistant server 200, there maybe a collection of nodes of approximately equal importance. For example,multiple users, each with their own assistant server 200, could connecttheir assistant servers together to function as a single assistantserver 200. An exemplary embodiment of this configuration is illustratedin FIG. 6B. FIG. 6B illustrates 4 instances 200 a-d, as well as mobiledevices 100 a-n in communication with the resulting assistant server200. It will be understood that any number of instances 200 a-n may beincluded. Furthermore, each instance 200 a-n may include some or all ofthe features of the assistant server 200 illustrated in FIG. 6A. In theconfiguration of the assistant server 200 illustrated in FIG. 6B,different instances 200 a-n of the assistant server 200 may be incommunication with all or some of the other instances 200 a-n. Asillustrated in FIG. 6B, one instance 200 a is in communication with allthe other instances, while another instance 200 b is only communicatingwith a single one of the others (200 a in this exemplary embodiment).Instance 200 b in this example will therefore only receive data directlyfrom the instance 200 a it is in communication with, but may receivedata from the other instances (200 c and 200 d in FIG. 6B) indirectly,e.g., through assistant server 200 a, which is in communication withthese other instances 200 c and 200 d. Any number of mobile devices 100a-n may communicate with the networked instances of the assistant server200 a-n. Each of the various mobile devices 100 a-n may each correspondto one of the instances 200 a-n. Furthermore, the various mobile devices100 a-n may each communicate with any of the instances 200 a-n which areconnected in this manner, to receive data which is shared between allthe instances 200 a-n. The instances 200 a-n may be “participants” in achat between users in this manner, similarly to services 300 a-nparticipating in chats as discussed above.

Alternatively, a single, “main” assistant server (instance) may functionas the assistant server 200 described above with respect to FIG. 6A, andalso connect to external nodes or instances for specific tasks andinformation. An exemplary embodiment of this configuration isillustrated in FIG. 6C. In FIG. 6C, only four instances 200 a-d areillustrated for convenience, but any number of instances 200 a-n may beincluded. In the exemplary embodiment illustrated in FIG. 6C instance200 a serves as the “main” assistant server, which mobile devices 100a-n are in communication with. This “main” assistant server is incommunication with other, “subordinate” instances 200 b-d. These“subordinate” instances 200 b-d are not necessarily communicatingdirectly with the “main” instance 200 a. As illustrated in FIG. 6C, forexample, instance 200 d communicates with instance 200 c, which in turncommunicates with the “main” assistant server 200 a.

As described above, icons, gestures, speech libraries, etc., are storedin the assistant server 200. However, some elements of these libraries,for example overriding aspects or additional aspects, could be containedelsewhere in the virtual assistant system 10, for example in theassistant interface 110 or in a node of the assistant server 200 that iskept separately from the other nodes. For example, a spoken command suchas “radiation start” may mean “switch the light bulbs on” for a specificuser, but may have a completely different meaning for other users. Thisadditional command may be in a node (the node comprising a CPU and amemory, serving basically as a server) that is separate from the mainassistant server 200 but connected to the assistant server 200, as in adeployment of multiple instances of the assistant server 200.

A user's personal access information, connections, sharing information,etc. might be contained in a separate instance of the assistant server200. This separate instance might be a server under the particularuser's control, such that the user may determine whether the separateinstance is connected to the assistant server 200. Use of this separateinstance enables a user to disconnect their personal information fromthe assistant server 200 and continue to be provided assistance by theassistant server 200, but at a diminished level without any personalinformation.

According to exemplary embodiments of the present general inventiveconcept, when a user wishes to perform a function with their mobiledevice 100, they must first run/initiate/open/access the Client 110 ontheir mobile device 100. Alternatively, the Client 110 may be running onthe mobile device 100 continuously, without need for initiation. Oncethe Client 110 is running, the user may make a command input, e.g.,select an icon to control an external service 300, in order to controlthe service 300 to perform a desired function. (Alternatively, theClient 110 may recognize a gesture, motion, sound, facial feature, orany other above-mentioned command input, without the need for a specificicon selection.)

When a command input selecting a service 300 is made and a functioncorresponding to the selected service is chosen via another or the samecommand input, the Client 110 may create a semantic atom 1000 thatincludes the command to control the service 300 to perform the desiredfunction, and send the semantic atom 1000 to the assistant server 200.The assistant server 200 may include a stored pre-programmed plugin 220that allows the assistant server 200 to translate the semantic atom 1000into a proprietary language of the selected service 300. The assistantserver 200 may receive the semantic atom 1000 from the mobile device100, recognize that the user desires to control a particular service 300in a particular way, and translate the semantic atom 1000 into theproprietary language of the service 300 according to the correspondingplugin 220 to allow the service 300 to understand the function/actionthat needs to be taken in response to the user's command input. Theassistant server 200 may then send the translated command in theproprietary language of the selected service 300 to the selected service300 in order to control the service 300 to perform the function.

For example, if a user wants to use a mobile device 100 to control alight (which may be connected with the virtual assistant system 10 viawired or wireless communications), that user may first select, e.g., alight icon displayed on a display screen of the mobile device 100 (forexample user interface 150), while the Client 110 is initialized. Thelight icon may open up command prompts or options to allow the user toselect, e.g., “LIGHT ON.” Upon such a user selection, the “LIGHT ON”command is converted into a semantic atom 1000 by the Client 110, and issent to the assistant server 200. This semantic atom 1000 may representthe concept of, e.g., turning on the light. The assistant server 200 mayreceive the semantic atom 1000. The semantic atom 1000 may then beconverted by the light plugin 220 within the assistant server 200 intothe light's proprietary language, the converted semantic atom 1000representing instructions understandable by the light. Morespecifically, a programmer may have previously programmed the plugin 220corresponding to the light to convert the device proprietary languageinto one or more semantic atoms 1000, and vice versa, thereby allowingthe user to control the light from their mobile device 100. As such, theassistant server 200 is able to interpret the “LIGHT ON” command sentfrom the mobile device 100, and then can send an instruction in thelight proprietary language to the light so that the light turns on.

As another example, if a user wants to use social media, e.g., Facebook™(which is programmed in a Facebook™ proprietary language) to findsomeone on, e.g., LinkedIn™ (which is programmed in a LinkedIn™proprietary language that cannot naturally communicate with theFacebook™ proprietary language), the user may open up the Facebook™application using the mobile device 100 that is connected with theassistant server 200. As such, the user uses the Client 110 to accessFacebook™, selects a person's profile in Facebook™, for example theperson's profile picture, and a semantic atom 1000 is created by theClient 110 to represent the selected person (i.e., the proprietarylanguage of Facebook™ is translated into a semantic atom 1000. Thissemantic atom 1000 may include any information retrieved from Facebook™,for example the person's name, pictures, and Facebook™ profile. Next,the Client may send the semantic atom 1000 representing the selectedperson to the assistant server 200, which accesses a stored plugin 220for LinkedIn™ in order to translate the semantic atom 1000 intoLinkedIn™ proprietary language. More specifically, the assistant server200 must have stored therein both a plugin 220 for Facebook™ and aplugin 220 for LinkedIn™, such that the semantic atom 1000 received fromFacebook™ representing the selected person can be used to find theselected person on LinkedIn™. In other words, the plugins 220 a-ntranslate proprietary languages of their corresponding services 300 a-ninto the single semantic language, and vice versa, in order to allowvarious services 300 a-n, including for example devices andapplications, to communicate with each other via semantic atoms 1000. Assuch, after the assistant server 200 translates the semantic atom 1000received from the Client 110 into the LinkedIn™ proprietary language,the assistant server 200 can directly send the instructions (in theLinkedIn™ proprietary language) to LinkedIn™ to perform the user'sdesired command, which, in this example, is to access the selectedperson's LinkedIn™ profile. Furthermore, the Client 110 may be operatingin the background, i.e., the user may not be consciously aware of theClient 110's operations or even the use of semantic atoms 1000. Insummary, as far as the user is concerned, the visual process of theabove procedure includes: (1) opening Facebook™′ (2) selecting aprofile, (3) opening LinkedIn™, (4) pasting the selected profile intothe search criteria in LinkedIn™, and (5) viewing the selected person'sprofile in LinkedIn™.

Additionally, the above operations may be distributed among services 300a-n. For example, a service 300 which has been given authorization tocommunicate in the semantic language, either by the software developeror via a command input from the user, may also communicate directly withthe assistant server 200 and mobile device 100 with semantic atoms 1000.The service 300 in this case effectively operates as its own plugin.Such an authorized service 300 would be capable of sending semanticatoms 1000 to other services 300, and performing other tasks of theClient 110.

Furthermore, a mobile device 100 that is connected with the assistantserver 200 (i.e., a mobile device 100 that includes the Client 110) canshare the semantic atom 1000 representing the selected person withanother mobile device 100 connected with the assistant server 200, inorder to allow the another mobile device 100 to use the semantic atom1000 representing the selected person to find the person's profile inyet another service 300, such as, for example, MySpace™. The sharing ofthe semantic atom 1000 may be performed via a messenger serviceproviding messaging or chat functionality, email, or any other type ofcommunication method involving the assistant server 200 and the mobiledevice(s) 100.

The above examples of service control are based on a service 300'scapability of having global connectivity. More specifically, in theabove examples, the assistant server 200 is the device that ultimatelycontrols the selected service 300, based on the commands received fromthe mobile device 100 via the Client 110.

Alternatively, there may be services 300 that only have localconnectivity capabilities, such as, for example, near fieldcommunication (NFC) devices. As such, control of such a localconnectivity device must initiate from the mobile device 100 directly,and the commands cannot be received from the assistant server 200 (forexample, the assistant server 200 is most likely out of NFC range of theNFC device, unless the assistant server 200 is included within themobile device 100, as in the exemplary embodiment illustrated in FIG.4B).

More specifically, when a user wishes to perform a function with theirmobile device 100 at a local connectivity service 300, the user mustfirst run/initiate/open/access the Client 110 on their mobile device100. Alternatively, the Client 110 may be running on the mobile device100 continuously without need for initiation. Once the Client 110 isinitiated, the user may make a command input, e.g., select an icon, tocontrol a local service 300 that is within a given range (e.g., an NFCrange/proximity) of the mobile device 100, in order to control the localservice 300 to perform a function. (Alternatively, the Client 110 mayrecognize a gesture, motion, sound, facial feature, or any otherabove-mentioned command, or the mobile device 100 may sense the localservice 300 automatically when within the given range without the needfor icon selection.)

When the command input is made and a function corresponding to theselected local service 300 is chosen, the Client 110 may create asemantic atom 1000 that includes the command to control the localservice 300 to perform the desired function, and send the semantic atom1000 to the assistant server 200. The assistant server 200 may include astored pre-programmed NFC device plugin 220 that allows the assistantserver 200 to translate/interpret the semantic atom 1000 into aproprietary language of the selected local service 300, and vice versa.The assistant server 200 receives the semantic atom 1000 from the mobiledevice 100, recognizes that the user desires to control the localservice 300 in a particular way, and uses the local service plugin 220to interpret the local service proprietary language commands that arenecessary to control the local service 300. Subsequently, the assistantserver 200 may create a new semantic atom 1000 including the commandsthat will be understandable by the local service 300, and send the newsemantic atom 1000 back to the mobile device 100.

The mobile device 100 may receive the new semantic atom 1000 includingthe commands that will be understandable by the local service 300, andtranslate the semantic atom 1000 into a command having the proprietarylanguage of the selected local service 300. Then, as long as the mobiledevice 100 is within the given range of the selected local service 300,the command (in the proprietary language of the selected local service300) is sent from the mobile device 100 to the local service 300 tocontrol the local service 300 to perform the desired function.

For example, if a user wants to use a mobile device 100 to control anNFC light (that is NOT connectable with the assistant server 200 viawired or wireless communications), that user can first select, e.g., alight icon displayed on a display screen of the mobile device 100 (forexample, user interface 150), while the Client 110 is initialized.Similarly to the example described above, selection of the light iconmay open up command prompts or options to allow the user to select acommand, e.g., “LIGHT ON.” Upon selection, the “LIGHT ON” command may beconverted into a semantic atom 1000 and sent to the assistant server200. This semantic atom 1000 may correspond to the action of turning thelight on. The assistant server 200 may receive the semantic atom 1000.The semantic atom 1000 may then be converted by the light plugin 220within the assistant server 200 into corresponding proprietary languagerepresenting instructions understandable by the light. Subsequently, theassistant server 200 creates a new semantic atom 1000 includinginstructions to generate the desired command(s) that will beunderstandable by the NFC light (i.e., expressed in the proprietarylanguage of the light), and sends the new semantic atom 1000 back to themobile device 100. Alternatively, the desired command(s) may already beconverted to the proprietary language of the light, and be included inthe semantic atom 1000 with instructions to extract the command(s)therefrom and transmit the extracted command(s) to the light to carryout the desired function(s).

The mobile device 100 may receive the new semantic atom 1000 includingthe commands that will be understandable by the NFC light, and convertthe semantic atom 1000 into a command having the proprietary language ofthe selected NFC device. This translation can be performed at theassistant interface 110, according to the instructions included in thereceived semantic atom 1000 to convert the atom 1000 to the proprietarylanguage of the light. Alternatively, the commands expressed in theproprietary language of the light may be extracted from the semanticatom 1000 at the assistant interface 110, according to instructionsincluded in the semantic atom 1000. Then, as long as the mobile device100 is within the NFC range of the NFC light, commands, e.g., thecommand to turn the light on, may be sent from the mobile device 100 tothe NFC light, in the proprietary language of the NFC light, to controlthe NFC light, e.g., to turn on.

Furthermore, if the light is password-protected so others cannot operatethe light without the password, an authorized user (i.e., a user who hasthe password or other credentials necessary to control the light) canshare the “LIGHT ON” semantic atom 1000 with an unauthorized user togrant the unauthorized user access to the light to perform the “LIGHTON” operation. In this case, the “LIGHT ON” semantic atom 1000 wouldfurther include the access credentials, e.g., password, user ID, etc.,necessary to access and control the light. The semantic atom 1000 caneither be shared via the assistant server 200 or directly from themobile device 100.

Alternatively, the user may grant the unauthorized user access to, e.g.,a service 300 temporarily by adding the unauthorized user's account as asemantic atom 1000 (created by the virtual assistant system 10) to alist of authorized users. As such, the service 300 never knows that anunauthorized user has been given access thereto, as the service 300continues to receive its commands from the virtual assistant system 10,notwithstanding whether an authorized user or an unauthorized user isproviding the commands.

This sharing of authority over services 300 is discussed in greaterdetail infra.

Additionally, the Client 110 may have incorporated logic therein toavoid the need to send the semantic atom 1000 to the assistant server200. More specifically, the Client 110 within the mobile device 100 maybe provided with a myriad of pre-stored plugins 220. The Client maytransmit and receive semantic atoms 1000 directly with these pre-storedplugins 220, without the need to route the semantic atoms 1000 throughthe assistant server 200. The Client 110 may also be pre-programmed withone or more applications authorized to use the semantic language andwhich may therefore communicate via semantic atoms 1000 without the needfor a separate plugin 220. Similarly to the pre-stored plugins 220, theClient 110 may transmit and receive semantic atoms 1000 with thesepre-programmed authorized applications without needing to route thesemantic atoms 1000 through the assistant server 200.

For example, if, as above, a user wants to use social media to findsomeone on, for example, LinkedIn™, the user may open, for example, aFacebook™ application using the Client 110 running on a mobile device100, select a person's profile (for example, the person's picture), anda semantic atom 1000 may be created by the Client 110 to represent theselected person. If plugins 220 corresponding to Facebook™ and LinkedIn™are directly stored within the mobile device 100, there is no need tosend the semantic atom 1000 representing the selected person to theassistant server 200. Instead, in this example, because the plugins 220are stored within the mobile device 100, when the user uses the Client110 to access LinkedIn™, the Client 110 translates the semantic atom1000 representing the selected person into the proprietary languageunderstandable by LinkedIn™. As such, the user can easily find theperson's profile on LinkedIn™ using the semantic atom 1000 correspondingto the selected person which was received from Facebook™, because thesemantic atom 1000 effectively allows Facebook™ to communicate withLinkedIn™. Moreover, the user's experience in this example issubstantially the same as when the plugins 220 are stored at theassistant server 200.

A process of determining the intention of an input command and carryingout the corresponding operation at an external service 300 will bedescribed while referring to FIG. 7A. Referring to FIG. 7A, when acommand is input at the mobile device 100 directly through the sensor160 or via an external sensor (operation S500), the assistant interface110 first translates the command input at the mobile device 100 into oneor more semantic atoms 1000 so that the command can be shared amongservices 300 a-n connected to the assistant server 200. As describedabove, the command input at the mobile device 100 may be an icon touch,gesture, voice data, or any other input provided at the mobile device100 in which a sensor part can receive the input.

The assistant server 200 receives this translated command (operationS510). The connection between the mobile device 100 and the assistantserver 200 may be automatic, or alternatively, the user may be promptedon the mobile device 100 to initiate and/or accept the connection to theassistant server 200.

The assistant server 200 then determines if the virtual assistant system10 is operating in a remote mode or a local mode (operation S520). Thedetermination of whether the virtual assistant system 10 operates inremote mode or local mode can include, for example, whether theassistant server 200 is permitted within the firewall of the service 300or if the assistant interface 110 is connected within a local network(such as Bluetooth or NFC) to the service 300.

If the assistant server 200 determines that the virtual assistant system10 is operating in remote mode (REMOTE at operation S520), the assistantserver 200 can confirm recognition of the command(s) originating at themobile device 100 by checking among the library of commands at operationS530 a.

If the received command is recognized at the assistant server 200 (YESat operation S530 a), the assistant server 200 can then perform thedesired command instructed (operation S540 a) at the intended externalservice 300 via the corresponding plugin 220 created at the assistantserver 200, when the virtual assistant system 10 is in the remote modeof operation.

Alternatively, if it is determined that there exists any confusion as towhat the command received from the mobile device 100 is intended torefer to (NO at operation S530 a), for example, there may be more thanone service 300 which uses the same command, the assistant server 200can examine the command in a number of contexts by considering, amongother things, the historical usage pattern of this command, theparticular user providing this command, the particular location in whichthe command is given, the specific time of day in which the command isgiven, whether this particular user providing the command always usesthis specific gesture to perform a specific function at a specificservice 300, etc., at operation S560 a. In other words, the assistantserver 200 will examine all accessible contextual information availablewithin a program running at the platform 210 of the assistant server 200regarding this received command in order to determine specifically whatpossible options could have been intended by the received command.

Once the assistant server 200 determines which possible options couldhave been intended by the command input at the mobile device 100, theassistant server 200 provides these options of intended commands back tothe mobile device 100 (operation S570 a) so that the options can bedisplayed at the user interface (UI) 150 at the mobile device 100. Herethe user can once again input the intended command, but this time withspecific accuracy by selecting one among a few displayed options. As anexample, a list of icons could be displayed on the mobile device 100representing the possible commands, and the user could select one iconas the intended command. The assistant server 200 may also record theuser's selection so that the original command input in operation S500will be recognized if it is input again later (a collection of historydata to form contexts). This is discussed in greater detail below withreference to FIG. 11.

The specifically intended command is then translated at the assistantinterface 110 (into one or more semantic atoms 1000) and provided to theassistant server 200, where the command is received at the assistantserver 200 (operation S580 a) to carry out this command in the remotemode of operation of the virtual assistant system 10. In other words,when the virtual assistant system 10 is in remote mode, at operationS540 a, the assistant server 200 may present the command to thecorresponding plugin 220, which converts the command from the one ormore semantic atoms 1000 to the proprietary language in which theexternal service 300 understands. The assistant server 200 then canperform the intended command, i.e., the assistant server 200 cancommunicate directly with the external service 300 in the respectiveproprietary language of the service 300 b to perform any functions oroperations instructed at the mobile device 100 (or other type sensordevice which is connected to the virtual assistant system 10 and canprovide a command input, which will be described in more detail infra).

Information received from the service 300 as part of carrying out thecommand in operation S540 a is converted from the proprietary languageto one or more semantic atoms 1000 at the corresponding plugin 220. Ifadditional operations are requested at other services 300 using thisinformation (YES at operation S550 a), additional commands input intothe mobile device 100 are received at operation S510 to perform theadditional commands.

Alternatively, when it is determined that the virtual assistant system10 is in a “local mode,” (LOCAL at operation S520), the virtualassistant system 10 can perform the following sequence of operations asillustrated in FIG. 7B.

If the received command is recognized at the assistant server 200 (YESat operation S530 b), the assistant server 200 can then obtain theproprietary language information of the specific service 300 to becontrolled, and provide the information to the assistant interface 110of the mobile device 100 as one or more semantic atoms 1000 with theproprietary language included (operation S535 b). With the necessaryproprietary language required to perform the intended control obtainedat the mobile device 100, control of the service 300 can be directlyperformed at the mobile device 100 through the assistant interface 110(and communicator 105) at operation S540 b. Furthermore, if additionaloperations are requested to be performed at other services 300 using theinformation obtained as a result of the function performed at thepreviously intended service 300 (YES at operation S550 b), additionalcommands input at the mobile device 100 are received at operation S510(FIG. 7A) to perform additional functions corresponding to theadditional input commands. These additional commands input can includeinformation obtained from the previous operations performed. Forexample, continuous communications between services such as “apps” oftenrequire information obtained from the previous “app” in which a functionwas performed (e.g., obtaining information on a movie playing in auser's locale).

If there exists confusion as to what input command may have beenintended when the command is received (NO at operation S530 b), theassistant server 200 then determines which possible options could havebeen intended by the command input at the mobile device 100 (operationS560 b). Similarly to the description above with regard to operationS560 a in FIG. 7A, this determination can be performed by examining thecommand in a number of contexts by considering, among other things, thehistorical usage pattern of this command, the particular user providingthis command, the particular location in which the command is given, thespecific time of day in which the command is given, whether thisparticular user providing the command always uses this specific gestureto perform a specific function at a specific service 300, etc.

Once the possible intended command options are determined at theassistant server 200 at operation S560 b, the assistant server 200 canobtain the proprietary language of these commands with the correspondingplugins 220, and provide these command options back to the mobile device100 (operation S570 b) with their respective proprietary languageinformation so that the options can be displayed at the user interface(UI) 150 of the mobile device 100. Here the actual intended command canbe selected and performed directly at the mobile device 100 (operationS540 b).

Similarly to operation S580 a described above, the assistant server 200may record the command option selected by the user in operation S570 bso that the command will be recognized if it is input again later.

As noted above, FIG. 7A illustrates a remote mode according to anexemplary embodiment of the present general inventive concept. In theremote mode, the assistant interface 110 does not need to be on the samenetwork or internet as the corresponding service 300, nor does theassistant interface 110 need to download the proprietary language of theservice 300. In remote mode, the assistant interface 110 may be requiredto present credentials such as a username and password to prove the userhas authorized access to the service 300, but communication usuallyoccurs through the assistant server 200.

Conversely, in the local mode illustrated in FIG. 7B, communicationoccurs directly between the mobile device 100 and the service 300. Inlocal mode, the assistant interface 110 does need to be on the samenetwork as the corresponding service 300, specifically the correspondinglocal network (for example, Bluetooth or NFC). However, once the mobiledevice 100 is on this local network, credentials are not necessary toprove the user has authorized access to the service 300.

In another exemplary embodiment of the present general inventiveconcept, the virtual assistant system 10 may operate in remote modewhile the assistant interface 110 communicates directly with the service300. This is advantageous if the service 300 monitors the number ofaccess requests and shuts off access to IP addresses that are accessingit excessively. For example, if one million (give or take) users sharingone assistant server 200 want to access the same service 300 through theassistant server 200, according to the control process illustrated inFIG. 7A, the service 300 would register a large number of accessrequests coming from one IP address (the assistant server 200). Theservice 300 may accordingly block access from the assistant server 200.In this situation, it would be preferable for each user to access theservice 300 through the assistant interfaces 110 a-n of their respectivemobile devices 100 a-n. The service 300 would then register one accessrequest from each of a million different IP addresses, and thereforewould not block access to any of the users.

The above exemplary embodiment of the present general inventive conceptis illustrated in FIG. 8. In this exemplary embodiment, the virtualassistant system 10 is operating in remote mode. That is, the commandsare being sent to a service 300 that is not connected locally (throughBluetooth, NFC, etc.) to the mobile device 100. In the exemplaryembodiment illustrated in FIG. 8, a user input is received at the mobiledevice 100 and translated into one or more semantic atoms 1000(operation S610). The input is then transmitted to the assistant server200 (operations S615), which in turn provides the translated input tothe corresponding plugin 220 (operation S620). Here the plugin 220converts this translated input to the appropriate proprietary language(operation S625).

After the input is converted to the proprietary language, a semanticatom 1000 including the command is sent to the mobile device 100(specifically to the assistant interface 110, through the communicator105, illustrated in FIG. 4A) (operation S630). The mobile device 100then extracts the converted command from the semantic atom 1000 andtransmits the converted command, in the corresponding proprietarylanguage, to the service 300 (operation S635). According to an exemplaryembodiment of the present general inventive concept, the convertedcommand may be transmitted by the assistant interface 110 through thecommunicator 105 over the Internet to the service 300.

The service 300 carries out the corresponding action and transmits databack to the assistant interface 110 (operation S640). This data is inthe proprietary language of the service 300, which the mobile device 100on its own does not understand. The mobile device 100 accordingly sendsthe received data to the assistant server 200 (operation S645), whichprovides it to the corresponding plugin 220 (operation S650). The plugin220 converts the data into one or more semantic atoms 1000 (operationS655). Similarly to the remote mode illustrated in FIG. 7A, once thedata is converted to the semantic atoms 1000, it may be shared betweenthe mobile device 100 and other services 300 a-n.

Furthermore, the present general inventive concept is not limited toonly using one of the local or remote mode at a time. For example, themobile device 100 could be connected locally to one or more services300, while also being connected through the assistant server 200 to adifferent service 300.

As described above, each plugin 220 created at the assistant server 200for the specific external service 300 is configured such that the plugin220 can convert commands in one or more semantic atoms 1000 into thecorresponding proprietary language. Similarly, the plugin 220 can alsoconvert this proprietary language and other information (i.e.,credentials of the service 300) into the semantic atoms 1000 used withinthe assistant server 200 so that the different external services 300 a-nconnected to the assistant server 200, via the Internet or other type ofwireless communication used, as described above, can now shareinformation between one another.

Furthermore, information obtained at the mobile device 100 from oneservice 300, as the result of an input command at the mobile device 100,can then be communicated and shared with other services 300 a-n via theassistant server 200 so that a user of the mobile device 100 cancommunicate information with and between each of the services 300 a-n.In other words, the mobile device 100 can control each of the services300 a-n and interact (share information) between the services 300 a-n toobtain a greater or more substantive bundle of information, which is alltransported as one or more semantic atoms 1000. This newly obtainedsubstantive bundle of information can then be used with another commandto be directed to another service 300 or shared with other mobiledevices 100 a-n connected with the assistant server 200. This processcan continue between external services 300 a-n and the mobile device 100as well as between mobile devices 100 a-n connected with the assistantserver 200 to share and obtain any information desired, and consequentlyto use this information to access any additional external service 300for an ultimate result sought.

An example of gathering information between services 300 a-n can be acase where a user of a mobile device 100 accesses a service 300 aFandango™ to determine which movies may be available in a givenlocation, and then choose a specific movie desired to view. Before theuser purchases tickets to this movie, the user can transmit or “toss”this movie information (translated into one or more semantic atoms 1000by the assistant interface 110) to another service 300 b, such as IMBD™.Then the user can determine which actors are in the movie via the IMDB™app prior to purchasing the tickets through Fandango™, based on whetherthe user may desire to watch a specific movie if the movie includes thisparticular actor.

In other words, once a movie is selected within the Fandango™ app(service 300 a), the assistant interface 110 can convert this movieinformation into one or more semantic atoms 1000, where this translatedmovie information can then be shared with other apps such as IMDB™(service 300 b) through the IMDB™ plugin in order to obtain additionalinformation about the movie. Such additional information to be obtainedcan include, for example, which actors are playing in the movie. IMDB™can receive and understand the movie selected within the Fandango™ appbecause this information is converted into one or more semantic atoms1000 from the Fandango™ proprietary language, and then can be convertedinto the proprietary language of IMDB™ by its corresponding plugin 220.

Furthermore, prior to purchasing the tickets, the user at the mobiledevice 100 may share one or more semantic atoms 1000 including thisobtained movie information with other users at other mobile devices 100a-n which are also connected with the assistant server 200 (to bedescribed in more detail with respect to FIG. 1A). Then the other usersmay view the obtained movie information and agree to a certain number oftickets to be purchased, and also may ultimately choose another movievia the same process, at which point all users can agree to purchasetickets for a movie which all users have agreed upon after sharing allof this information gathered between Fandango™ and IMDB™, etc., andbetween the users of the connected mobile devices 100 a-n themselves.The above example is only an example of shared communication betweenusers and services 300 a-n, and is not limited thereto. Any number ofcommunications and any format of communications between services 300 a-nand users (via mobile devices 100 a-n) can be performed through thesemantic atoms 1000 created within the assistant server 200.

Another example of gathering information between services 300 a-n can berelated to music. For example, the mobile device 100 can use apps suchas Gracenote™, Shazam™, or any other type of music recognition softwareto recognize music that is playing somewhere (such as a restaurant,shopping mall, music store, radio, television, etc.). The musicrecognition can be performed by a microphone type sensor within themobile device 100 or connected to the mobile device 100. The songrecognized by the assistant interface 110 can then be converted into oneor more semantic atoms 1000 and sent to the assistant server 200 topresent this information at the Gracenote™ app through aGracenote™-created plugin 220. Gracenote™ can then provide the songinformation to the Gracenote™ plugin 220, which in turn will provide thesong information as one or more semantic atoms 1000 to be used for anyadditional purpose that may be requested by the user at the mobiledevice 100.

In other words, at this point any additional service 300 that knowsand/or understands how to use a song and is connected to the assistantserver 200 can use this music information (as one or more semantic atoms1000) to provide any desired result, as may be requested by the user atthe mobile device 100. For example, this information can be sent, viathe same process, to the app MusixMatch™, which presents the lyrics forthe song. It can also be sent to Rdio™, which adds the song into theRdio™ queue for the user. It can also be sent to Sonos™ speakers, whichplay the song. It can be also sent to YouTube™, which can find a videoof the song and play this video or add the video to the user's playliston YouTube™. The number of apps that this obtained and expandedinformation can be sent to in order to perform a command or gather evenmore information is only limited to the number of apps or other servicesthat can work with perform some type of function using this obtainedmusic information.

In each case, the plugin 220 for each service 300 desired to be usedwith this obtained music information can translate to-and-from thesemantic atoms 1000 and the proprietary language of the service 300,thus providing for the free flow of communication regarding the musicinformation between all services 300 a-n, as well as between multipleusers. Furthermore, the music information provided in the semantic atoms1000 can be sent from one user at a mobile device 100 to other users ofother mobile devices 100 a-n to allow multiple users to apply this musicinformation to various services 300 a-n of their choosing.

In addition to the above described capabilities, if a song is playing ona television, for example, and a user of a mobile device 100 desires tocapture, save and play the song on an external portable MP3 player orother type of music playing device at a later time, the user can firstuse Gracenote™ to recognize the song playing on the television.Subsequently, this song can be translated to one or more semantic atoms1000 by the assistant interface 110 and provided to the assistant server200, where the translated song can then be sent to a plugin 220 createdfor the external portable MP3 player, where the translated songinformation can be translated to the proprietary language of theexternal portable MP3 player, where the external portable MP3 player canthen be instructed by either the plugin 220 itself or the mobile device100 to play the song at the user's convenience. Whether the plugin 220is instructed to directly play the song on the external portable MP3player or whether the plugin 220 provides the proprietary language andother necessary information back to the assistant interface 110 so thatthe mobile device 100 can play the song on the external portable MP3player can depend on either the network in which the MP3 player isconnected, the location of the MP3 player with respect to the mobiledevice 100, or in accordance with other conditions as describedpreviously herein. As such, the virtual assistant system 10 can accessany Internet-based webpage or plugin 220 and use one or more semanticatoms 1000 to perform operations and interconnectivity with otherInternet-based webpages, plugins 220 a-n, mobile devices 100 a-n, orexternal services 300 a-n.

An example of using atoms 1000 to communicate between services 300 a-nis illustrated in FIG. 9. As illustrated therein, a user can use thevirtual assistant system 10 to, for example, look at posts made bypeople in social media, for example Facebook™ (corresponding to service300 a). The virtual assistant system 10 may represent each person with acorresponding semantic atom 1000. The user can take an atom 1000 whichrepresents a person and ask another service 300 b to provide additionalinformation on the person, which may be added to the semantic atom 1000.For example, the user can send the atom 1000 to, e.g., Gmail™, toretrieve the person's email address. The e-mail address so retrieved canbe added to the semantic atom 1000 that represents the person. The usercan use this email-enriched semantic atom 1000 to send the person anemail on another service 300 c, for example Yahoomail™. Moreover, theuser can use the virtual assistant system 10 to access a person's résumévia another service 300 d, for example LinkedIn™ and add this additionalinformation about that person to the atom 1000. The atom 1000 cancontinue to be used and added to, with an infinite number of services300 a-n. In the example given here and illustrated in FIG. 9, if theperson's résumé includes a musical band that the person belongs to, forexample, then the user can use the semantic atom 1000 representing theperson (including their résumé and musical band noted thereon) to accessan Internet web page such as Billboard™ (another service 300 e) in orderto view songs written by the person. Similarly, the user could use thesemantic atom 1000 to directly access an application such as Pandora™(service 300 f) to play random music written by the person.

As such, semantic atoms 1000 allow a user to instantly connect aconcept, e.g., a person, to various types of applications and services300 a-n. Several examples of the use of semantic atoms 1000 are providedherein:

With respect to movies, a user may use the virtual assistant system 10to, for example, look at the top movies currently available accordingto, e.g., the Rotten Tomatoes™ service. Each movie may be represented bya separate semantic atom 1000. The user may use a movie's semantic atom1000 to locate the movie for rent, for example to verify which localRedbox™ have the DVD in stock. Any command input or combinations thereofmay be used, depending on the particular user and the programming of theplugins 220 involved. For example, no typing or speaking may benecessary. This process of looking up a movie and finding itsavailability may all be accomplished purely with touches or gestures.Similarly, Netflix™, Blockbuster™, Crackle™ or any other type ofmovie-related services may be utilized in this example.

With respect to books, a user may find a book on a service 300 a, forexample Goodreads™ and obtain a semantic atom 1000 corresponding to thebook. Then the user may send this atom 1000 to another service 300 b,for example Amazon™ to see people's ratings for the book. Subsequently,the user may send the semantic atom 1000 to, e.g., Overstock.com™(another service 300 c) to purchase the book. As such, the user caninstantly use the semantic atom 1000 to purchase the book and maintain arecord of the purchase on the virtual assistant system 10.

With respect to light bulbs, a user may, for example, generate asemantic atom 1000 representing the color and brightness setting of abulb for bulb settings. Alternatively, two separate semantic atoms 1000may be made for color and brightness, which may be combined to form asingle semantic atom 1000. Then, the user may “throw” the atom 1000representing the bulb's settings at another bulb using the virtualassistant system 10 using, e.g., a gesture movement. The other bulb maybe immediately forced (i.e., commanded in its proprietary language) toadopt the color and brightness settings of the first bulb.

With respect to cross-user sharing, semantic atoms 1000 can also be sentto other users. As illustrated for example in FIG. 10, one user canrecognize a song using a music recognition service 300 a, e.g.,Gracenote™ which may obtain information about the song, such as the nameand album. The virtual assistant system 10 may generate a semantic atom1000 representing the song and including the information found by theservice 300 a. The user may then send this semantic atom 1000representing the song to another user, who may then use the song atom1000 to play the song on, e.g., YouTube™ via the virtual assistantsystem 10, by sending the atom 1000 to another service 300 b (YouTube™in this example). This other service 300 b may obtain the song and playit on the other user's audio system. Notably, the assistant server 200in this example is optional for transmitting the song informationbetween users. A user having a semantic atom 1000 representing a song ontheir mobile device 100 may send that atom to another user's mobiledevice 100 a, directly or via the assistant server 200.

Furthermore, as described above, several atoms 1000 may be nested insideone another or bundled (combined) into a single atom 1000. For example,a virtual shopping basket full of shopping items (for example, a user'scart on Amazon™) can be a single atom 1000 that includes other atoms1000 corresponding to the individual shopping items nested inside it. Assuch, the single atom 1000 of the shopping basket can save space andincrease the convenience of transmitting the atom 1000 (and its nestedatoms 1000) between different services 300 a-n to increase userconvenience.

The virtual assistant system 10 is an open one in which any developercan create a plugin 220. This open construction allows new semanticatoms 1000 to be specified by any developer for new data types. In thisway, the semantic language may keep expanding over time. Semanticlanguage dialects are also permissible for different domains.

In addition to providing a common set of commands, exemplary embodimentsof the present general inventive concept also allow new commands to bedeveloped, via an adaptive learning and control system. For example, acommand received at the mobile device 100 which is not recognized may berecorded as a new command once defined by the user. As illustrated, forexample, in FIG. 11, a user may make a gesture intended to give acommand (“Stop”, as illustrated in FIG. 11). If this input is not amongthe inputs recorded in the assistant server 200, the user may then inputthe command a different way, for example by touching an icon denoting“Stop.” The assistant server 200 may then record the first input (thegesture in this example) into the storage 250 as denoting the associatedcommand.

According to an exemplary embodiment of the present general inventiveconcept as illustrated in FIG. 12, natural and involuntary inputs canalso be interpreted as commands by the virtual assistant system 10. Forexample, if a user of the virtual assistant system 10 plays a video andthe audio is too loud, the user may involuntarily flinch and jerkhis/her head back. This involuntary motion will be followed by the useof icons or gestures or voice commands to lower the volume. The virtualassistant system 10 may thereby learn that the flinch and head-jerk-backgesture should be associated with lowering the volume by a large amountor muting it altogether. This new command (flinch and head-jerk-back)may be accordingly saved in the storage 250 of the assistant server 200.Similarly a grimace or other facial expression in certain contexts couldbe associated with ‘undo or reverse the last command’. As anotherexample, if a user squints at a display screen (e.g., user interface150) of the mobile device 100, this facial expression could beinterpreted as a command to enlarge a displayed text or image.Accordingly, the virtual assistant system 10 can adapt to and storeunconscious or involuntary input commands in addition to conscious inputcommands.

As an alternative, or in addition to the above exemplary embodiment ofthe present general inventive concept, a user or a developer couldspecify specific gestures which represent specific people. As anespecially contrived example, snapping one's fingers, tapping onesforehead and then gesturing to indicate a clockwise movement followed bytwo palms open to indicate ‘10’ might mean ‘send my mother a messagethat I will be there in 10 minutes.” Hence, some inputs may have a rangeof variance that represent specific things like numbers or individuals.

Some variance may also be provided in the inputs, such as the icons andgestures, themselves. Therefore, several icons may be specified in thelibrary which are mutually interchangeable and equivalent. Similarly,several gestures could correspond to the same underlying meaning and beinterchangeably used. This interchangeability of command inputs isimportant because different cultural zones may have varying meaningsassociated with specific things and have preferences which are varying.Note that the color and size of icons could also have meaning associatedwith them. Similarly, icons and gestures could take on additionalmeaning when used in conjunction with one another, such as denotingwhich service 300 is being controlled. For example, an icon representing‘close’ could be used to manipulate a door closed when the right palm isheld flat out but be used to close a window screen when a U shape ismade with right hand instead. Such rules would also constitute a part ofhow the library is predefined and then used by plugin developers toprovide users with controls. Note that gesture repetition may also carrymeaning. Making one knocking action with a finger versus two knocks inrapid succession could be entirely different gestures in the gesturelibrary. As another example, for a plugin developer who is connectingthe virtual assistant system 10 to a document management system,pinching fingers (as though holding an imaginary key) and turning oncemight mean make a document ‘read only’ whereas turning it twice couldmean ‘remove both read and write access.’

Another example of such adaptive use according to an exemplaryembodiment of the present general inventive concept is illustrated inFIG. 13. As described above with respect to FIG. 4A, the mobile device100 may include a Central Processing Unit (CPU) 130. This CPU 130 may beconfigured to run a learning algorithm which determines when a series ofcommands are customarily given together. For example, a user may make afirst command, illustrated as a gesture, to raise a thermostattemperature to 70° F. The user may then make another gesture to turn onliving room lights to maximum brightness, and a third gesture to turn onthe living room television. If the CPU 130, running the learningalgorithm, determines that these gestures are customarily performedtogether, it may communicate this determination to the assistant server200, which in turn may generate a new command which executes all threeactions simultaneously. FIG. 13 illustrates a separate new gesture toexecute the group of commands customarily presented together. In otherexemplary embodiments of the present general inventive concept, multipleequivalent command inputs (for example, gestures) may be proposed, anyone of which will execute all the associated actions. When one of thesecommand inputs is detected at the mobile device 100, it is translatedinto one or more semantic atoms 1000 by the assistant interface 110 andtransmitted to the assistant server 200, where it is then translatedthrough the plugins 220 to give the appropriate command to thecorresponding services 300 a-n.

In other exemplary embodiments of the present general inventive concept,the virtual assistant system 10, specifically the mobile device 100, maydetermine that specific icons customarily receive touch inputs at aspecific time of day, and then bring these specific icons to moreprominent positions on the user interface 150 at that time of day. Thevirtual assistant system 10 may also present these icons (for example,as a visual depiction) on the user interface 150 of the mobile device100 in more prominent positions compared to other icons at that time ofday. The virtual assistant system 10 may alternatively generate a newicon at the assistant server 200 and displayed on the mobile device 100which executes the actions of several icons customarily receiving touchinputs together. These correlations are saved—in human readable ornon-readable format—in the storage 250 attached to the assistant server200. The virtual assistant system 10 will generate a name for this iconbased on the context. For example, an icon could be named ‘arrivedhome’.

The virtual assistant system 10 may be adaptive in another respect. If,via the CPU 130 running the learning algorithm, the mobile device 100notices that the user incorrectly but repeatedly uses an icon orsequence of icons with the intention of executing an action which failsto be executed (because the icons were incorrect) then the assistantserver 200 may actually associate that icon or icon-sequence with thataction (for that user, for a subset of users, or for all users). Forexample, if a user repeatedly tends to turn up the volume and thenimmediately turns it back down and then (using a different icon) movesup a channel of a television, this may indicate that the user isconfusing the icon for turning up the volume with channel-up. The CPU130 in the mobile device 100 can detect this behavior and then proposethat the “volume-up” icon be used for “channel-up” instead and in turnbe replaced with a different and less-confusing icon. The virtualassistant system 10 can also automatically make the change if it deemsit appropriate based on a decision-making algorithm that is running inthe CPU 130, the assistant server platform 210, or on a separate serverwhose results are saved on the storage unit 140 or storage 250.

These usage patterns and the accompanying changes may be collected inthe storage unit 140 of the mobile device 100 or in the storage 250 ofthe assistant server 200 to be used as updated commands. As a result,the library of commands 230 stored on the storage 250 may expand overtime with new, intuitive and adaptive commands.

Alternatively, or in addition to the above adaptive control in which theassistant server 200 automatically updates the commands, data on theuser's actions and inputs may be collected and made available to theplugin developer, for example, by e-mail or an online administrativeportal through which the developer can view information related to theusage of the plugin 220. This allows the developer to update the plugin220 accordingly. This is valuable because the repeated use of theinputs, for example icons, indicates a persistent intuitive mapping ofthe inputs and the intended action in the mind of that user or in anaggregate group of users. The developer can discover that such apsychological input-to-action mapping exists based on user behavior.Separately transmitting the data to the developer instead ofimplementing a learning algorithm in a CPU can be advantageous whenprocessor capacity or the effectiveness of a particular learningalgorithm is a concern.

These adaptive techniques may be employed to address semantic confusionbetween plugins 220 created by different plugin developers. For example,the developer of the plugin 220 for one thermostat may specify onegesture to raise temperature whereas the developer of the plugin 220 fora different thermostat may specify a different one. Based on how usersare employing gestures, the virtual assistant system 10 may providesuggestions to the developer on how gestures (or icons or voicecommands) can be modified to allow users to accomplish what they wantwith a more consistent set of gestures. By classifying services intosimilar groups, developers will also be helped to select the mostintuitive icons, gestures and voice commands for their new plugins 220.For example, a developer creating the plugin 220 for the tenththermostat to interface with the virtual assistant system 10 may be ableto be better informed by how users actually prefer to invoke variousthermostat functions by means of icons, gestures and voice with theprevious nine thermostats. This could be done prescriptively by stronglysuggesting the developer use specific icons for specific functions ormore indirectly by discouraging mappings which have been shown to beconfusing for users in the past.

Furthermore, these adaptive techniques can be employed using variousartificial intelligence techniques, which are to be utilized by and/ordeveloped on various types of hardware, including robotics algorithms,learning algorithms, parallel programming, logical searching,optimization, heuristics, evolutionary computation, etc., but are notlimited thereto. Accordingly, the virtual assistant system 10 can“learn” a user's preferences based on the user's movements, habits, andtendencies.

According to another exemplary embodiment of the present generalinventive concept, different sets of gestures, icons and voiced commandscan become established for different groups of users (e.g. in differentcountries). The virtual assistant system 10 (either the CPU 130 of themobile device 100 or the assistant server platform 210 of the assistantserver 200) can infer from the use of specific inputs (gestures, voicecommands, icons, etc.) which of these groups the user belongs to andthen provide support to them in the “dialects” that user is mostfamiliar with. For example, it is possible that people coming frommilitary contexts get used to a different set of icons or gestures tomean “turn left” than people coming from civilian contexts. If thevirtual assistant system 10 is able to infer that someone ismilitary-affiliated based on a gesture the person makes to indicate“turn left,” other gestures and icons may also change meaning to providethis person with a more familiar context. Such inferences can then besaved in the virtual assistant system 10 as a context of this person(specifically, in the storage unit 140 of the mobile device 100 and/orthe storage 250 of the assistant server 200).

According to another exemplary embodiment of the present generalinventive concept, there may be a three tier system of libraries. Thefirst set of (gesture, voice, icon) libraries are universal and apply toall users. The second set of (gesture, voice, icon) libraries apply tospecific cohorts of users (e.g. those in a specific trade, those of acertain age or social grouping, those who speak a given language, thosewho are members of a specific set of institutions, those who reside inspecific regions or countries, etc.). The third set of (gesture, voice,icon) libraries apply to specific users. If there is conflict betweenthese because the same command inputs (e.g., words, gestures, icons,etc.) mean different things in these three different tiers of libraries,a precedence rule can be utilized. Normally, user-specificinterpretations trump user-cohort interpretations which in turn trumpuniversal interpretations. However, this precedence order can bedifferent in some situations.

Controls may also be automated or vary depending on context. Forexample, if the user always turns on the DVR after turning on the TV,the virtual assistant system 10 can automatically choose to turn on theTV when receiving a command to turn on the DVR. Alternatively, thispredictive behavior may be programmed by the plugin developers accordingto use or context data gathered by the virtual assistant system 10. Asanother example, access to a certain document management system canautomatically be enabled if a user is present in a particular geo-fencedarea, and such access may be cut off when the user exits that area.Control can be made even more specific, e.g., as illustrated in FIG. 14,a user that enters a room through one corridor they might have access tothe document management system, whereas if the same user enters from adifferent corridor that user may not have access. Other factors, forexample, time or the sequence of user usages of the virtual assistantsystem 10 can also be an element of context, thereby varying thecontrols of the virtual assistant system 10.

In order to enable these scenarios where the virtual assistant system 10takes predictive action, according to an exemplary embodiment of thepresent general inventive concept of the virtual assistant system 10,the mobile device 100 or the assistant server 200 can constantly measurewhen certain actions are invoked and collect these measurements in itsstorage (storage unit 140 of the mobile device 100 and/or the storage250 of the assistant server 200). The measurements could includelocations, times of day, co-presence of specific individuals, actionsequences that have been taken and other factors. If the virtualassistant system 10 finds a reliable way to predict when certain actionsare taken based on statistical, probabilistic or machine-learningalgorithms that are running on the CPU 130, the assistant serverplatform 210, or on servers or other learning type devices connected toor in communication with the virtual assistant system 10, then thevirtual assistant system 10 can do so.

The virtual assistant system 10 can also proactively present itsintention to predictively take such actions to the end user by means ofdisplaying a message on the mobile device 100. This message can betransmitted from the assistant server 200 to the mobile device 100 to bedisplayed on the UI 150, or the message can be generated at the CPU 130and displayed at the UI 150. The message may be, e.g., visual, audible,or tactile, and allow the user to approve or disapprove such actions.The message can take the form of presenting the predictive action inadvance as a new “standing rule” or proposing that the virtual assistantsystem 10 take the action momentarily unless the user overrides theproposal and instructs the virtual assistant system 10 not to.

In another exemplary embodiment of the present general inventiveconcept, the mobile device 100 is configured to provide a virtualreality (VR) experience. In this case, the mobile device 100 may be,e.g., a wearable device such as a headset or gloves. The use of semanticatoms 1000 may enable a seamless VR experience.

VR in this context denotes an experience in which the user interactswith an immersive environment. An “immersive” environment means onewhich partially or completely replaces a user's perception of theirsurroundings with a virtual environment. For example, the mobile device100 may be a headset worn by the user and which presents a view acrosspart or all of the user's field of vision. The user of such a mobiledevice 100 perceives an artificial view of a virtual environment throughthe mobile device 100. As such, the user can be immersed in the virtualenvironment. Alternatively, the user may be in an enclosed environment,e.g. a room in which images are displayed on the walls, and the “mobiledevice 100” is stationary objects which may receive input commands fromthe user, e.g., by receiving gesture and voice commands. In other words,a user may have a VR experience without necessarily being required towear anything directly on their body while still being immersed in anartificial environment.

The present general inventive concept enables VR by providing a unifiedenvironment for a user to interact with. Similarly to services 300 a-nparticipating in a chat through plugins 220 a-n, services 300 a-nparticipate in a VR session through plugins 220 a-n, with the userinteracting with the plugins 220 a-n. For example, upon initiating a VRsession, the user may enter an initial VR environment of their choice.This first environment may correspond to a first plugin 220 a. From thisinitial environment, the user may transit seamlessly between other VRexperiences via other VR environments, where each different VRenvironment may correspond to one or more different plugins 220 b-n.Relevant information regarding the user and command inputs may becommunicated between plugins 220 a-n via semantic atoms 1000. As such,semantic atoms 1000 streamline communications between different services300 a-n providing different VR experiences, allowing different VRexperiences to be integrated with each other and operate togetherseamlessly.

User commands may still be accepted through the medium of the VRenvironment, for example through icon touches, gestures, or spokencommands. User command inputs may be interpreted contextually, e.g.,through interaction with the VR environment. For example, a virtual iconmay be displayed in the user's field of vision, and a touching motion bythe user on the icon may correspond to a command to perform anassociated action. A “touch” in this case may mean a gesture thatlogically equals a touch except that the user is “touching” somethingbeing displayed to the user in a VR display as opposed to somethingbeing displayed on a screen or other object the user is actuallyphysically touching.

As noted above, the mobile device 100 including assistant interface 110which translates a command input or inputs into one or more semanticatoms 1000 may be, for example, a VR headset or other wearable devicethat the user uses as part of the VR session. The mobile device 100 mayalternatively be a separate computing device in communication with theheadset or other wearable device, similarly to FIG. 5B. The assistantinterface 110 of such a mobile device 100 may translate command inputsreceived from external sensing devices such as the headset into one ormore semantic atoms 1000, and may transmit the translated commands tothe assistant server 200.

In addition to allowing for seamless transition(s) between differentservices 300 a-n making up the VR experience, exemplary embodiments ofthe virtual assistant system 10 allow cooperation between multipleusers. Multiple users may participate in a VR session, experiencing thesame or similar VR environments. Each user may use a separate virtualassistant system 10, including their own mobile device 100 and assistantserver 200. The various assistant servers 200 a-n may communicate witheach other via semantic atoms 1000. Alternatively, an individual usermay connect their mobile device 100 to two or more assistant servers 200a-n, which may similarly communicate with each other with semantic atoms1000. Furthermore, different users may all use the same assistant server200, even if the users are physically distant from each other, e.g., indifferent countries or time zones, so long as their respective mobiledevices 100 are in communication with the assistant server 200.

When there are multiple instances of the assistant server 200,information such as user preferences, command inputs, and contexts maybe communicated between assistant servers 200 a-n, allowing oneassistant server 200 a to understand the command inputs of anotherassistant server 200 b. This is a form of adaptive learning by theassistant server(s) 200, described in detail above. For example, commandinputs of multiple different users can be communicated between andunderstood by the multiple instances 200 a-n of the assistant server.Multiple instances 200 a-n of the assistant server also enable theexample given above in which multiple users are using completelydifferent assistant servers 200 a-n, but are still participating in thesame VR experience, as their respective assistant servers 200 a-ncommunicate with each other.

In a shared VR experience between two or more users, command inputs mayalso be shared. One user may make a command input that can then becorrected or complemented by another person. For example, one of theusers might make a gesture to change a scene, which might fail becausethat user is unaware of the correct gesture to make (as registered inthe library 230 of stored command inputs accessed by the assistantserver 200, illustrated in FIG. 6A). Then, a second user might correctthe first attempted command input, for example by means of a gesture, aspoken command or by touching icons, which is recognizable as a gesturewithin the library 230 of command inputs accessed by the assistantserver 200. The assistant server 200 may then update its library 230with the original, non-recognized command input, and associate theoriginal command input with the intended action. This updating of thelibrary is also an example of adaptive learning, which enables theassistant server 200 to adaptively expand its vocabulary of commandinputs. This same action can be extended to multiple instances of theassistant server 200, as different instances 200 a-n of the assistantserver may communicate the new command input to each other to updatetheir respective libraries of command inputs.

Command inputs from multiple users can also be bundled together viaadaptive learning. For instance, if some actions repeatedly are beingperformed on a sequence of gestures, voice statements or virtualizedtouches of icons or graphics, the virtual assistant system 10 can bundlethese actions together into a single gesture, voiced command orgraphical icon touch that will trigger that entire sequence of actionsto be performed. It is also similarly possible to adaptively bundletogether actions that involve multiple people. For example, if someaction is taken by User A, then followed by an action by User B and thenanother by User C, but these users often perform these actions inconjunction together, all these actions can be bundled together into asingle trigger (e.g., a gesture, voice control, or graphic control),which can be made by any of the users. Sometimes, these actions can alsorequire authorization from multiple people together or in sequence. Ifso, these authorizations can also be bundled together into a singlecontrol by extending the relevant authority conditionally for thatbundled action alone. This bundling of actions relates to the extensionand revocation of authority described in detail infra.

The meaning of command inputs may vary depending on context. Forexample, if a gesture command input is made in the direction of a light,the virtual assistant system 10 may interpret the gesture as, e.g., acommand to reduce the brightness level. If the same gesture is made inthe direction of a music player, the virtual assistant system 10 mayinterpret the gesture as, e.g., a command to reduce a volume level. Thisform of adaptive learning is extremely valuable in VR applications, toallow the virtual assistant system 10 to perform more intuitive actionsin response to command inputs.

The same command inputs may also be interpreted differently fordifferent people. For example, if one person consistently tends to use agesture to mean one thing and another person uses the same gesture tomean something different, the virtual assistant system can learn thesepreferences via adaptive learning. Furthermore, if multiple instances200 a-n of the assistant server are used by one or more users, theinstances 200 a-n of the assistant server may communicate userpreferences and command inputs between one another in order to support aseamless experience for one or more users. Note that this use ofadaptive learning to interpret command inputs differently for differentusers also permits the emergence of dialects of command inputs indifferent user groups, or between individuals.

Equivalences may be established between handheld experiences (e.g. witha smartphone) and VR experiences for specific command inputs. So, in thelibrary 230 of command inputs, some command inputs may be marked for usein handheld experiences, some command inputs may be marked for VRexperiences, and some command inputs may be marked for both handheld andVR experiences. In some cases, there will be a mapping where a specificmobile command input (for example, an icon touch) corresponds to adifferent VR command input (for example, a gesture). These mappings canbe inferred by the virtual assistant system 10 based on usage andfeedback patterns, via adaptive learning. The mappings may also beexplicitly provided by plugin developers.

Plugin developers may have the ability to design experiences (e.g.define one or more actions for plugins 220 a-n) without defining whichspecific commands should cause these one or more actions to be performedat the corresponding service 300. Plugin developers can then laterdefine the relevant command input(s) by mapping control elements (e.g.,voice, gesture, touches, etc.) onto the defined one or more actions.Alternatively, or in addition, the virtual assistant system 10 mayrecommend one or more specific command inputs to use in specificsituations based on experiences with other plugins 220. These commandinputs can also be fine-tuned, as well as adapted, based on userbehavior. As an example, a plugin developer for a music player plugin220 can program relevant actions, e.g., change volume, change musicbeing played, etc., while developing the plugin 220. The plugindeveloper may then later program the relevant command inputs, forexample, a thumbs-up to increase volume, an icon touch gesture to changemusic, etc. Alternatively, the plugin 220 may be provided to the virtualassistant system 10 without specific command inputs defined, and thevirtual assistant system 10 may suggest or provide command inputs. Forexample, if the virtual assistant system 10 has adaptively learned fromother plugins 220 a-n a specific gesture that a user typically uses toraise volume, the virtual assistant system 10 can suggest or supply thatgesture to the music player plugin 220 in the above example tocorrespond to the command input to raise volume.

In some cases, the control elements that come to mean certain actions byplugins 220 or by the virtual assistant system 10 will also be adaptedbased on subconscious user behavior. If the user consistently does somethings unconsciously, the virtual assistant system 10 may adapt to theuser based on that behavior. For example, if a user consistentlyattempts to move sideways in a street-view type virtual walk through acity, the virtual assistant system 10 might observe this attempt to moveand make such sideways motion a supported gesture corresponding to thataction. This interpretation of command inputs can also be contextdependent. For example, gestures that correspond to one or more controlsfor one or more services when user A is accompanied by user B mightcorrespond to one or more different controls for one or more serviceswhen user A is instead accompanied by user C. Gestures that correspondto one or more controls when a user is in a citywalk plugin 220 mightcorrespond to one or more different controls when the same user is usinganother plugin 220 to virtually fly over an oceanscape. Control elementscan also have different meanings within a single plugin 220, fordifferent internal contexts within the plugin 220. For example, for theplugin 220 providing a virtual walk through a city described above, acommand input (e.g., a gesture or movement of the user's feet) may causesideways movement when the user is on a virtual street, while the samecommand input may cause the user to enter a virtual building if the useris near a virtual building in the same virtual walk plugin 220. In thisexample, the virtual location of the user in the VR experiencecorresponds to a context used to interpret one or more command inputs.

The Client 110, corresponding to the user's interface with the virtualassistant system 10, can be personified in some way in a VR environment,and the user may be able to interact with it, e.g. talk or gesture tothe client (consciously or unconsciously). For example, the Client 110may be represented as a bird that the user perceives in the VRenvironment, and which the user can interact with directly, e.g., togive one or more command inputs and/or receive feedback from services300. Presented in this fashion, a particular user's Client 110 may beperceivable only by that user, or alternatively may be perceivable byone or more other users participating in the same VR experience. Auser's Client 110 can also be present and monitoring the user, for alimited duration or permanently, without being personified or evenperceivable by the user at all. In other words, the user may not beconsciously aware of their Client 110's presence in a VR session,although the Client 110 is still monitoring the user's actions andaccepting command inputs based on the user's actions.

VR also allows for intuitive clarification of command inputs. When auser performs a command input that is not understood or may beincorrectly understood by the virtual assistant system 10, the user canresort to a graphical system, for example a display of selectable icons,that can help the system adapt and learn the command inputs that werenot understood.

It is possible that different users who are present together in a commonVR session may have different views or experiences of the samesituation. For example, a user might pick a movie from their movieservice 300 a (for example, Hulu Plus™) and share the movie with anotheruser (for example, by making a command input to control the virtualassistant system 10 to share the movie with the other user). The otheruser, receiving the shared movie, may have a different movie service 300b (for example, Netflix™) and could check the availability of the sharedmovie in this service 300 b.

In this situation, data corresponding to the shared movie may berepresented as one or more semantic atoms 1000. This data (the movie inthis case) may then be mapped between services 300 a-n, through thecorresponding plugins 220 a-n. A user can also share data expressed asone or more semantic atoms 1000 with another user. The other user canthen translate this data into their respective services 300 a-n easily.

It is also possible that the very session environment that two or moreusers are sharing is constructed using such transfers of data viasemantic atoms 1000. For example, the specific color setting(s),scenery, audio background(s) and other aspects of the experience mightbe constructed by multiple VR systems exchanging semantic atoms 1000 viathe assistant server 200, which then composes the information includedin the atoms 1000 together into a single session environment. For thispurpose, there could be multiple instances of the assistant server 200running in the different users' VR systems, e.g., each user may havetheir own assistant server 200 a-n, in communication with one or moreother users' assistant server(s) 200 a-n in the manner described above.

There are also scenarios where in a single VR session with one user, theuser is interacting with the virtual assistant system 10 and moving dataexpressed as semantic atoms 1000 across services 300 a-n. For instance,the user might look up an article of clothing on Pinterest™ (through aplugin 220 a corresponding to Pinterest™) and then use that article ofclothing to find and preview similar items on Nordstrom™ (through aplugin 220 b corresponding to Nordstrom™) with ease. The user might evenbuild a collection of items on one service 300, for instance, and thenthat list could get distributed across several other services 300 a-n.For instance, over some period of time on Rotten Tomatoes™ a user mightassemble a list of movies they wish to watch. This list of movies couldthen be distributed across movie services, e.g. Netflix™, Hulu Plus™Amazon Instant™ and other services, based on the availability of moviesin the respective services 300 a-n. Such lists of items (movies in thisexample) can also be collaboratively built by multiple people workingtogether.

Semantic atoms 1000 are also important in order to ensure seamlessexperiences as users exit one plugin 220 (corresponding to a VRexperience designed by one developer) and enter another plugin 220(corresponding to another VR experience designed by another or the samedeveloper). With a conventional electronic device, for example asmartphone, a user transitions between services 300 a-n (e.g. apps onthe smartphone) by consciously stopping one service 300 a, returning toa list of available services 300 a-n (e.g. a “home screen”) andselecting a new service 300 b to run. While this form of interface isacceptable for conventional electronics, it does not lead to animmersive VR experience. According to the conventional approach, a userin a VR session who wants to change between services 300 a-n, forexample different VR environments, must stop one VR session and start anew session with the new environment. This transition can break thecontinuity of a user's VR experience, since the user must stop onevirtual reality experience completely in order to start another virtualreality experience. Furthermore, in starting a new VR session in a newVR environment, the user frequently must adjust to the new environment,e.g., establish their preferences and settings all over again. Thisadjustment can also break the user's sense of a continuous virtualexperience. Exemplary embodiments of the virtual assistant system 10 mayaddress this problem. Rather than force a user to step out of a VRexperience in order to switch between services 300 a-n, the virtualassistant system 10 described herein may allow a user to transitseamlessly between services 300 a-n.

To establish such a seamless experience, the user's context and otherelements which are relevant to the user's experience in one plugin canbe shared with or provided to other plugins 220 as one or more semanticatoms 1000. For instance, a user who is walking down a virtual street inVR according to a first plugin 220 a can climb into a virtual spaceshipthat is positioned on the virtual street, having been provided to thefirst plugin 220 a by another, second plugin 220 b corresponding to thespaceship, in the form of the semantic atoms 1000. Data from the firstplugin 220 a can be expressed as semantic atoms 1000 and shared with orprovided to the second plugin 220 b. Once the user has virtually climbedinto the virtual spaceship, the user can then control the virtualspaceship in VR, for example, taking off, and the user has then actuallytransitioned to using the second plugin 220 b, while all being performedseamlessly. In the user's perception, the user has had a continuous VRexperience in which the user has walked down a virtual street, climbedinto a virtual spaceship, and taken off in the virtual spaceship. Thiscontinuous VR experience is possible because the virtual assistantsystem 10 can manage the interactions between plugins 220 a-n viasemantic atoms 1000. This use of semantic atoms 1000 is especiallyimportant with respect to managing a user's VR experience.

In fact, there could be several VR experiences a user may look at anypoint within a plugin 220 providing a VR experience, and each one ofthese VR experiences may correspond to VR experiences provided bydifferent plugins 220 a-n. When the user enters one of these VRexperiences, any relevant data may be shared with or provided to the newplugin 220 a as one or more semantic atoms 1000. This data may also beassociated with user authority which may be extended to another plugin220 b, as well as with payment instruments which may travel with theuser, i.e. the authority to use the payment instruments (e.g. chargetransactions to the payment instruments) may accompany the user as theuser moves between VR experiences. This transfer of relevant databetween plugins 220 a-n allows a user to combine VR experiences,provided by one or more plugin developers, into a single, seamless VRexperience.

Data expressed as semantic atoms 1000 can also contain information abouta user's current status, e.g. with respect to rewards programs or theuser's achievement of some kind of status. Atoms 1000 representing dataabout the user's status can then be shared between or provided to otherplugins 220 a-n easily. Each plugin 220 will have its own definition ofwhat a particular status means. Sharing or providing data about userstatuses allows users to build on past experiences between plugins 220a-n. For example, one or more users may move their intensive play in afirst VR game to a second VR game, to give the one or more users ahigher starting point in the second VR game.

If data represented as one or more semantic atoms 1000 is stored in thevirtual assistant system 10, it is also possible that people engage inVR sessions at disparate times but are able to use data that has beenpreviously obtained from services 300 a-n by other users. These usersmay then provide further data in the form of one or more additionalsemantic atoms 1000.

Exemplary embodiments of the present general inventive concept may alsoinclude augmented reality (“AR”). AR is related to VR, in that the useris presented with a virtual environment. The primary difference betweenAR and VR is that the user may still perceive their immediatesurroundings, only additional graphics or other overlays may beprovided. For example, a user trying to follow directions to adestination may perceive an arrow on the ground, guiding the usertowards the destination. It will be understood that the inventiveconcepts presented herein are equally applicable to AR as to VR.

An AR experience may be provided, for example, through glasses thatallow the user to see normally but also display images in the user'sfield of view. Alternatively, or in addition, the user may be watchingfor example a real-time or time-delayed camera feed, in which additionalelements are displayed for viewing. An AR experience may also provideadditional data to one or more users' other senses, for example viaaudio signals or haptic feedback.

Much like VR, the interpretation of command inputs and the actionsperformed in AR may depend on the accompanying context. In AR, howeverthe contexts can include the user's immediate environment. As such, thevirtual assistant system 10 may include or be connected to, for example,cameras and other sensors that detect the user's surroundings and allowthe virtual assistant system 10 to react thereto. As an example, ifthere is an intruder in the user's home, a camera of an ARexperience—such as a camera included in the glasses described above—maycapture an image of the invader's face, and report it to theauthorities. This image capture and report function may be performed inresponse to a conscious or unconscious action on the part of the user.

As will now be described in more detail while referring to FIG. 1A andFIG. 15, a user of the mobile device 100 may share access to theassistant server 200 (and hence the virtual assistant system 10) toallow other users to control services 300 a-n within the virtualassistant system 10 and therefore already being controlled by the usersharing the access. Thus, a user (at mobile device 100) can sharesemantic atoms 1000 with other users through the other users' respectivemobile devices 100 a-n. Accordingly, any desired control of a service300 by the user of the mobile device 100 can be shared with another userthrough the other user's mobile device 100 a (see dotted lines “sharingaccess”) by “friending” the other user's mobile device 100 a, asillustrated in FIG. 1A. This sharing (or “friending”) of access tocontrol services 300 a-n can be provided once the user (mobile device100) has a way of contacting the other user, for example, via email,LinkedIn™, Facebook™, Twitter™, etc. In other words, once a user of themobile device 100 within the virtual assistant system 10 is able tocommunicate with another user via another user's mobile device 100 a-nthrough any form of communication, the user (mobile device 100) can sendan “offer” to share access to the assistant server 200 with the otherusers' mobile devices 100 a-n via this form of communication.

Sharing in particular may be enabled through messaging, in which a usermay request access from another user and be granted access through areply message. These messages may also include atoms 1000 representing,e.g., the access granted.

Once the other user(s) accepts the offer of access to the user'sassistant server 200, the other user(s) may be provided with the samecontrol over all external services 300 a-n as the user of the mobiledevice 100 possesses through the assistant server 200. Moreover, asdescribed in more detail below, the shared access with other users maybe limited in many aspects, such as, for example, with respect to thelevel of control, the location in which the other user(s) is situated, atime frame in which the user allows other user(s) to maintain the sharedaccess, etc. There are a vast number of ways in which access sharing toother users may be controlled/limited based on the desired setup of thevirtual assistant system 10 by either the developer of the system 10 orthe user's instructions at a later time after the original setup of thesystem 10.

The shared access allows the other users to control external services300 a-n through the original user's assistant server 200 and to thelevel at which the original user allows the shared (or “friended”) userto have access.

Furthermore, as illustrated in FIG. 15, at #1, although the originaluser A (the user making the offer) must initially provide ausername/security ID and password (or “access credentials”) in order toaccess an external service 300, the user(s) B who becomes “friended”(receive shared access) does not need to enter this security ID orpassword. Therefore the original user A is never required to share theseaccess credentials with the other user(s) B being “friended.”Accordingly, when user A instructs the assistant server 200 to allowaccess to a service 300 (see #2 of FIG. 15), the user B receiving theshared access can then control the service 300 a without knowing theaccess credentials input by user A, as pointed out at #3 of FIG. 15.Thus, no access credentials need to be exposed or shared with any otherusers in order to “friend” another user(s) and allow the other user(s)to have a full or limited access to any service 300, or even theassistant server 200. Furthermore, the original user A can instruct theassistant server 200 to revoke user B's access to the service 300 at anytime, or alternatively can declare specific conditions under whichrevocation of access to other users, i.e., user B, can occur.

In addition to keeping the ID and password (or other credentials)confidential from other users, since the ID and password remain only atthe assistant server 200, and therefore never need to be communicatedwirelessly, the user's IDs and passwords (or other credentials) cansafely remain confidential to the original user A (at mobile device100). Accordingly, illegal access, impermissible access and misuse ofsuch credentials via a wireless transmission interception or other typeof compromised security can be entirely prevented.

Thus, the assistant server 200 maintains all credentials (ID, passwords,etc.) that give authority to access and use the underlying services 300a-n. These credentials are maintained by the assistant server 200 at thestorage 250 that is connected either directly or indirectly to theassistant server 200. When a plugin 220 is created at the assistantserver 200, and a user begins to access this plugin 220 for the firsttime in order to have the assistant server 200 carry out various actionswith the corresponding service 300, the user first needs to provide therequired credentials for the service 300 to the assistant server 200 insome form. These credentials can be collected by the assistant server200 in one of many means: through a screen, keyboard, scanner, mouse,camera, microphone or hand-held or wearable device (such as armband orfacewear or fingerwear, etc.) or another peripheral hardware system orsystems connected to or in communication with the assistant server 200.

As an example, the user A (at mobile device 100) may provide a usernameand password for an email service to the assistant server 200. Now theassistant server 200 becomes capable of accessing and using the servicevia the Internet, Bluetooth, Optical or visual signaling, audiosignaling (through an audio speaker or other mechanism), tactile orother mechanical signaling using a device such as a vibration unit,WiFi, USB, FireWire, HDMI or through other means on the user's behalf.The user can now instruct the assistant server 200 to share access tothe service 300 with another user(s) B as well. As described above, thisinstruction by user A enables another user(s) B to access the service300 via the assistant server 200, and only through the assistant server200. As far as the underlying service 300 (e.g., email) is concerned, itis being accessed by the original user (user A) either through theassistant server 200 or through the mobile device 100 if the assistantserver 200 and the mobile device 100 are provided together in onehardware, and only the assistant server 200 is aware that the service300 is, in fact, being accessed by another user (user B).

Another example of this secured form of sharing is described as follows,while still referring to FIG. 15: the original user A establishes aservice 300 in the assistant server 200 using the assistant interface110, e.g., the original user A may establish a certain set of lights asan external service 300 to be accessed and controlled wirelessly. Theuser may then provide the assistant server 200 necessary informationrequired to access and operate the lights. This information couldinclude, for example, a password, username, and an IP address of thelights. The plugin engineer initially specifies which credentials areneeded for a user to start using that type of external service 300 withthe assistant server 200. Now the lights exist as the external service300 in which the assistant server 200 can access. This access control isowned only by that user A. This user A can now instruct the assistantserver 200 to “friend” those lights (service 300) to any other user B asdesired. This sharing of access can be accomplished in various differentways as described herein. For example, user A may instruct the assistantserver 200 the share access by contacting user B and offering user B tojoin the virtual assistant system 10, or can make a gesture toward theuser B if user B is within an area in which a gesture can signal user B,or user A can make a gesture that signifies allowing access, at whichpoint a camera detects a user B in the vicinity, etc. The other user(s)B can now see the external service 300 via the assistant server 200 anduse the external service 300. The other user B can, for example, turnthese lights on and off, and change the color of these lights. However,as described above, this other user B does not possess the username,password or IP address (or other credentials as originally created bythe engineer) of the lights and has no means of operating the externalservice 300 except through the assistant server 200.

At any time the owner user A can instruct the assistant server 200 to“unfriend” the friended user B (revoke access) from the external service300. If the owner user A revokes access to the other user(s) B, theother user(s) B ceases to be able to operate the lights or, in somecases, user B may even be blocked from being able to see these lights inany fashion at all. That is, user B's mobile device 100 a may be blockedfrom even being aware of the lights.

A user may also set criteria by which “unfriending” occursautomatically. Such criteria can be a time limit of access shared withuser B, a location condition placed on user B, and so on. For example,services may be shared with the other user B for a preset amount oftime, or until the other user leaves a particular area.

An alternative exemplary embodiment of sharing access involves tossingsemantic atoms 1000 between different users. For example, if a user isin a library in France that requires a password and credit payments foraccess to the WiFi, the user can send a request from a mobile device 100a (for example, the user's tablet computer), to all the user's friends(within Gmail™, Facebook™, MySpace™, etc.) to verify whether any of theuser's friends have ever been in that library, whether they accessed theWiFi, and whether they have remaining payment credits that they are ableto share. Then, the user can send a request to one of the friends foraccess to the WiFi in the library in France, even if the friend is in adifferent country. The friend can then either provide the WiFi accessmanually by “tossing” a semantic atom 1000 including their password toaccess the WiFi to the user. Alternatively, the friend could have presetthe access to be provided upon request, i.e., preset tossing thesemantic atom 1000 with the password to the user. In either case, theuser receiving the WiFi access never needs to know the password toaccess the WiFi, as the information is sent as a semantic atom 1000 fromthe friend to the user. The received semantic atom 1000 can appear as anicon, attachment, image, file, email, chat, etc., but is not limitedthereto, and allows the user to automatically access the WiFi uponreceipt of the semantic atom 1000, or manually access the WiFi uponselection of the semantic atom 1000, without having any knowledge of theWiFi password.

FIG. 16 illustrates an exemplary embodiment of the present generalinventive concept in which access to a service 300 may be shared betweendifferent assistant servers 200, for example if there are multipleinstances of the assistant server 200. As illustrated in FIG. 16, User Ahaving a first assistant server 200 a may share access to a service 300with User B having a second assistant server 200 b. The access controlinformation (e.g., username, passwords, etc.) is contained in the firstassistant server 200 a (specifically, storage 250 a corresponding to thefirst assistant server 200 a) under User A's control. If access isshared with User B, User B's assistant server 200 b may access theservice 300 through User A's assistant server 200 a, and the accesscredentials and plugin 220 stored thereon. Furthermore, User A canspecify whether they want User B to continue to have access should thefirst assistant server 200 a be turned off or otherwise disconnectedfrom assistant server 200 b. If User A determines that the access shouldcontinue, the access information stored in assistant server 200 a, aswell as the corresponding plugin 220, may be mirrored in assistantserver 200 b (specifically, in storage 250 b corresponding to assistantserver 200 b), but then removed should User A revoke access to theshared service 300. Similarly to the above examples in which there isone assistant server 200 shared by two users, when this access controlinformation is mirrored in User B's assistant server, User B is unawareof the details of such access control information. For example, User Bdoes not know User A's username and passwords to access the sharedservice(s) 300.

There are an extensive number of other applications that can beimplemented by the virtual assistant system 10. Controlling a set ofhome services 300 a-n are one obvious application. If someone visits auser's home, the host (the user of the virtual assistant system 10) caninstruct the virtual assistant system 10 to give the guest courtesyaccess to all or a large subset of the connected services 300 a-n in theuser's home (lights, thermostat, window blinds, alarm system, phonesystem, WiFi hub, entertainment systems, etc.) as long as the usersbeing “friended” are within the physical confines of the home. Thiscould also be done based on calendar invites. In other words, along withinviting others to a user's home, the user owning the home could alsocreate a formal invite to the visiting users to be “friended” whichspecifies access to home facilities during the time of the visit. Thisformal invite could be sent via any form of messaging, such as, forexample, e-mail, text message, etc. Messaging shared access in thismanner is a form of “actionable messaging.” The atom 1000 correspondingto the shared access (including, for example, access credentials) may besent as a message to another user, potentially with other messagecontent, e.g., text inviting the receiving user to the sharing user'shome.

This limited access sharing can also be implemented in a businesscontext. When someone joins a team, for example, the virtual assistantsystem 10 can extend access to various tools and services to this personvia the assistant server 200. When this person ceases being part of theteam, this access can be revoked. This revocation can be performed bysimply requesting a list of those members provided with access, and thenselecting among certain members in which the user requests access to berevoked.

The assistant server 200 can also provide this access based on paymentsbeing made. For example, a visitor to a movie theater can have access toa soda fountain via the theater's assistant server 200 for as long as amovie is playing. This time based access sharing can be automaticallyrevoked at any designated time, i.e., at completion of the movie.

As another example, a user A (of mobile device 100) can shareinformation from all his/her health devices with his/her doctor. Thedoctor, being user B, can be provided access to all health informationstored in the storage 250 of the assistant server 200.

As described above, the shared access offered by the initiating user Aof the mobile device 100 to other users B can be completely controlledby the user A of the initiating mobile device 100 such that access byany of the shared access users B can be terminated at any time by theowner user A, as well as being limited to a certain level of control, acertain time period of control, control limited to within a certainarea/location, etc.

Sharing can also be predictive in nature, where the predictive rules andalgorithms may or may not be humanly readable, and are saved in astorage 250 connected to or in communication with the assistant server200, where they are running. Alternatively, as pointed out above, undercertain situations the predictive rules and algorithms related to theaccess sharing can be running on the mobile device 100, and thereforestored in the storage unit 140. For example, if a certain set ofcontexts usually result in a user A sharing access to some services withanother user B, the assistant server 200 may recognize this contextualpattern and in the future perform this sharing automatically. Theassistant server 200 may also present the owner user A with the choiceas to whether the contextual pattern should be determined and appliedautomatically, or whether the user A should make this decision. Forexample, the assistant server 200 can alert the user A that theassistant server 200 intends to share access to the lights or otherservices 300 a-n based on a recognized contextual historic pattern, suchas sharing a thermostat control, a television, etc., with a guest user Bunless the user A explicitly wants to refuse such access. At this pointthe user A can select an option provided by the assistant server 200 bytouching an icon on the display 150, making a gesture to a sensor 160within the mobile device 100, and provide an input command via anotherform of input as described herein.

As pointed out above, sharing is also possible to perform at varyinglevels. For example, an online commerce service like Amazon™ hasmultiple things one can do with it. Ratings and pricing of various itemscan be reviewed in a personalized way (“this is the rating for otherreaders like you” or “as a prime member, you don't pay for shipping forthis item”), items can be added to a wish list, items can be added tothe Amazon™ shopping cart and purchases can be made. It is possible tocreate subset services where items can be reviewed and placed in theAmazon™ shopping cart but purchases cannot be completed. It is possibleto see how a parent (user A) can provide their children (user B) withsharing an Amazon™ account to the extent of putting things in theAmazon™ shopping cart but not actually making the purchase. This way theparent (user A) can review all items the children (user B) want beforethe purchase is made. In other words, access sharing can apply to evensubsets of a plugin's functionality rather than all of itsfunctionality. Similarly, other rules can also be crafted. A parent canlimit his/her children to change TV channels within certain values,while the parent (User A) can go to any channel desired via theassistant server 200.

This limited and context-based sharing has other applications also. Forexample, a city can allow an average citizen to operate doors or othersystems in emergency situations where it is normally prohibited to them,and can only be operated by a specialist or normally-authorizedpersonnel. Access can also be extended to specific groups of people.Based on these people being added to a specific user group, pregnantwomen (in a business, hospital, shopping mall or community), forexample, may be given access to certain services 300 a-n which areinaccessible to others for the duration of their pregnancy. Users whoseelectric cars break down in a specific way, for example, may have accessto loaner cars in unmanned car parks where any normal user does not.Access to the user's broken-down car may predictively be granted to arepairman who heads out to the car. Here, the users A, via the mobiledevice(s) 100, will extend specific permissions to the repairman'smobile device 100 a.

Social, family and community connections will be inferable from sharingpatterns as well. This too can be used to make predictive sharinghappen. If all home devices are being shared by one user A with anotheruser(s) B, it is likely that the next home device to be registered atuser A's home will also be shared with the same other user(s) B. Suchpatterns will be saved in the storage 250 and will be inferred usinganalytic algorithms created within the assistant server platform 210 orobtained as patterns that can be imported into the assistant serverplatform 210 from other software programs running on other servers.

Note that users can quickly forget their usernames and passwords tovarious services where access via the assistant server 200 suffices. Anowning user A of mobile device 100 can also designate another user B ashaving the power to share access with yet other users. This can beimplemented, for example, among members of a single household. One usercan purchase a TV and interface it with the assistant server 200, andthen provide access rights to other family members (users B), who canthen each extend access to this TV (service 300) as-needed to theirrespective guests.

The assistant server 200 can also transfer ownership of services fromone user A to another. This allows, for example, user A to sell theirhouse to another person and transfer all services 300 a-n within thehouse to that person. Similarly, a person (user A) changing offices cantransfer ownership of all services 300 a-n from themselves to anotheruser B, where user A can then take ownership of other services 300 a-nassociated with a new office in which user A moves into. As pointed outabove, services can be granted based on certain conditions also. Forexample, if a person unfortunately deceases, then their services 300 a-ncan automatically be turned over to their relevant family members orother desired people.

As pointed out above, gestures are a form of commands used with thevirtual assistant system 10. Gestures are an easy way to operate theauthority sharing features described above. The use ofgestures—including eye movement or non-movement, eye brow movement, agaze and a facial expression and all other visually-detected bodymovements (or non-movement) to extend or revoke authority of usage tousers or groups of users—can be quite powerful, and can be performedusing the virtual assistant system 10 as described herein. However, itwill be understood that, as noted above, any other command input, forexample, icon touches or spoken commands, may be used in place ofgestures.

Referring to FIG. 17, sharing access to control of a set of lights(external service 300) to a user B could be as simple as the owner userA pointing to the lights and then making a gesture representing sharing(see Time #1) this access with the other user B. As a result, theassistant server 200 can grant access to user B. More specifically, insuch an environment where a user A regularly works or lives, forexample, a set of sensors, such as a camera or other type ofphoto-detecting device that are capable of capturing information about auser's movements, can be set up within the controlled environment. Thesesensors can be configured to have their own form of an assistantinterface 110 a-n such that when these sensors (see FIG. 17) detect agesture by user A to share access to control services 300 a-n such aslight bulbs, etc., within the controlled environment, this gesturecommand can be translated at the relevant sensor into one or moresemantic atoms 1000 and provided directly to the assistant server 200.As a result, the assistant server 200 can share access with a user B asinstructed by the user A via a gesture command input representingsharing access. As noted above, FIG. 17 also illustrates an exemplaryembodiment of stationary “mobile” devices 100, with the various sensorshaving assistant interfaces 110 a-n.

For higher security situations, a camera or other type image sensor orother type sensor may be able to detect the actual hand or fingerprintsof the user A in order to determine that it is in fact user A that isinstructing the assistant server 200 to share access to another user B.Alternatively, the user A may have a wearable type device oraccelerometer that detects a gesture made by user A. Moreover, when thesensor detects user B via the camera or other type recognition device,the assistant server 200 can store any predetermined detected charactertrait of user B and store this information within the storage 250 sothat shared access may be provided to user B during later visits to suchan environment.

Still referring to FIG. 17, revoking control to a user B can be assimple as making a gesture for “unshare” towards user B (see Time #2),which leads the assistant server 200 to revoke access to the device touser B. Alternatively, revoking can be performed by other gestures, suchas, for example, making the same set of gestures as those of sharingaccess to user B, and then making a chopping gesture. This is merely anexample to illustrate the various types of gestures that can beprogrammed into or learned by the assistant server 200. These gesturesby user A can also be captured by a camera or other photo-detecting orsensing device that is capable of capturing information about movementsof user A, which is in turn connected to the user's mobile device 100.Such gestures may also be captured by wearable, held or other devicesthat detect movement by using accelerometer hardware or othermovement-tracking hardware, or even eye-tracking, lip-tracking orfacial-tracking systems.

Alternatively, as described previously, user A may request a recent listof users B in which access has been shared. Once this list is providedto user A at the user interface 150 of mobile device 100 by theassistant server 200, user A may choose any user(s) B on the listprovided at the user interface 150 and instruct the assistant server 200through the user interface 150 to revoke future access.

It is also possible to set-up rules where access is extended temporarilyto another user B and then must be extended periodically (or upon theoccurrence of an event, such as someone departing from a geo-fencedarea) with the use of a gesture. In this type of access sharing, if theowning user A makes such a gesture then access can be continued. If theowning user A fails to make the gesture, then access lapses.

Gestures may also be involuntary or be made in a way of which a user Bis unaware. For example, a commercial establishment where products aresold may make door locks to certain restrooms operate for customers orprospective customers (i.e., user(s) B) who spend specific amounts oftime in certain areas of the establishment, but not to others. Such acommercial establishment may even allow goods contained in securedclosets or display cases to become accessible to certain customers basedon gestures and movement information, which are not available to others,by granting them access to open the securing enclosures where others arenot granted such access.

In geo-fencing and other such applications, locations can be determinedby means of a GPS unit, WiFi unit, Bluetooth facility,location-broadcasting buoys, IP addresses, NFC, etc., which areconnected to or in communication with the assistant server 200.Proximity-detection to location measuring sensing devices which may beembedded in automobiles, household objects, office objects and otherobjects can also measure location. Time can also be an element of theaccess determining means which can be setup and measured by a similarsuch computing device. The determining means of access can also includethe sequence of user usages of the assistant server 200.

The virtual assistant system 10 described herein also provides theability for users to have secret gestures for revoking and grantingaccess to control external services 300 a-n. For example, in a homebreak-in situation, a parent User A may grant the right to a child tosummon the police using the home alarm via a secret gesture (where thechild may not normally have this right).

These gestures may also operate in conjunction with icon-touches andvoiced commands. The user A, for instance, might make a gesture thatmeans ‘share’, then user A may recite the names of all the serviceshe/she intends to share. A voice sensing device can detect the namedservices and accordingly provide access for these services. Then user Acan point at the receiving user B as described above.

These gestures may also be used in situations where a subset of usersfrom a large subset of users have to be granted or ungranted access to aservice (or set of services). An example of such a situation is where adecision needs to be made as to who has access to a given cash registerin a store from among a hundred sales associates. In such a situation,simply laying out a list of names or photos of the associates andpointing at the particular ones to whom access should be granted can beone mechanism for granting a subset of users particular access. Anothermechanism can be through gaze-tracking where the user A looks at theparticular ones who should be granted provisional access longer than theamount of time user A looks at the ones who should not be grantedaccess. Such gestures can be conscious or unconscious on the part ofuser A.

Similarly, a situation can occur in the reverse where there are a largeset of services and a decision needs to be rapidly made as to whichservices access should be provided (or access revoked from) for a useror a set of users. Gestures by the user A—conscious or unconscious—canbe used in this situation as well.

Gestures may also be used to seek access to services 300 a-n by usersseeking shared access (i.e., a user B seeking access in which the user Awith access rights). This enables a “silent” conversation where one userseeks access to a given service 300 with one set of gestures and isgranted such access—either by an owner user A with another set ofgestures or through some decision made programmatically(algorithmically) by the assistant server 200 and sensor devicecombination which is observing the gestured request. These algorithmsthat determine such shared access can be software programs or parts ofsoftware programs running as part of the assistant server 200 togetherwith other programs running within the assistant server platform 210.

In an exemplary embodiment of the virtual assistant system 10 providinga virtual reality experience, a user may share authority to control oneor more services 300 with other users. Similarly, a user can choose toshare authority over different aspects of their virtual realityexperience with one or more other users for the duration of the VRsession, for other lengths of time, or subject to other conditions. Forexample, two or more users can participate together in a game(corresponding to a service 300 a) where one user has the authority toplay the game and extends that authority to other users for a limitedtime basis. As another example, multiple users can shop together usingthe authority (corresponding to a shopping account) of one of the usersin a virtual shopping experience to interact and buy from severalmerchants (corresponding to one or more services 300 a-n).

Sharing authority can be very important in business-related VR sessions.For example, a meeting could be held in a VR session where theparticipants are reviewing an asset, e.g. a document. The owner of thatasset can choose to extend viewing authority to some participants andediting authority to others. The owner can even choose to have differentlevels of authority (e.g., viewing authority, editing authority, etc.)be presented to the different participants which can be revoked at anytime of the owner's choosing.

One important distinction virtual reality will often have from otherenvironments is that it may be quite social and often involvecollaborative scenarios. Often authority will need to be pooled to makesomething happen. That is, users can concurrently share their respectiveauthority with each other, so that the users are all granted the sumtotal of the authority. For example, there could be games wheredifferent users have paid for different segments of a game (or multipleinterconnected games) but can extend the authority to play in thosesegments to one another. In those situations, the virtual assistantsystem 10 can allow the authority to be pooled, thereby allowing everyuser to play the authorized segments of the game. As discussed above,sometimes combined actions can be constructed that involve authorityfrom multiple users also.

Authority trades are also possible, where the virtual assistant system10 arranges a simultaneous swap of authority over some underlying asset,for example the authority to issue commands to one or more services 300a-n, permanently or temporarily, based on an agreement between two ormore users. These authority swaps can be simple trades between twousers, but complex trades can also be arranged between more than twoparties. In the same way, a user may permanently transfer the authorityover an underlying asset to another user without receiving anything inreturn, or in exchange for a transfer of payment. Following such atransfer of authority, the original user loses their authority over theunderlying asset. In this way, sales or transfers of access or authoritycan be made in virtual reality. For example, a user selling their homecan transfer their authority over the home's thermostat to a buyer. Thevirtual assistant system 10 can also hold things (e.g., authority toissue commands to one or more services 300 a-n) in escrow to ensure thetransaction is completed properly.

A messaging session between two or more users may itself be a contextfor sharing authority. For example, a user in a messaging session canindicate to the virtual assistant system 10 that they are extendingcontrol over something (e.g., a service 300) to one or more other userswithin the medium in which the messaging session is being performed.This control could be temporary, permanent, or revocable upon userinstruction or upon a certain context, e.g., upon the completion of amessaging session. For example, two users in a group chat session maydesire to play a game of chess, using a service 300 that only one of theusers has access rights to. The user with access rights could extendthose access rights to the other user for the duration of the chatsession, so that both users would have access rights to use the chessservice during the chat session. As another example, a user can add awebcam view (corresponding to an output from a service 300) into a chatsession which is only operational as long as the chat session lasts.This webcam view may not be directly related to the participants in thechat session, e.g., it may be a view different from that of aconventional video conference session. For example, the webcam view maybe a view from a movable webcam (service 300) focused outward on aphysical space that can then be manipulated by the users in the chatsession to swivel or zoom. The owner of this webcam may extend authorityto the other users in the chat session to control the webcam, e.g., byviewing the output and manipulating the view, without being required toalso share access criteria, such as, for example, usernames andpasswords.

A messaging session can further be a context used to limit the extensionof authority. For example a presentation might be made available by auser to other users of a messaging session only as long as thatmessaging session lasts. Similarly, a user might be able to view acamera in a chat session with another user (or set of users) but not beable to view the camera anywhere else. Or the users may be able to viewa presentation that has been shared in messaging when those users aremessaging with the owner-user, but not at another time without theowner-user. Similarly, there may be services (for example songs ormovies) that two or more users can use together inside a messagingsession when one user is authorized, i.e., when one user has authorityover the services, but have access to those services be denied toun-authorized users when such users are not accompanied by an authorizeduser. As another example of contextual extension of authority, a usermay share authority for one or more services with members of a chatgroup as long as the user is a member of the chat group. If thatauthorized user leaves the chat group, any authority extended by thatuser to the other users in the chat for the one or more services mayautomatically lapse. This contextual extension of authority allowsmultiple users to pool authority. For example, a group of users may poolauthority for a list of movies that the users individually have accessto via a particular service 300 a. When a user leaves that group, thatuser's authorized movies would no longer be accessible by the otherusers in the group.

Contextual extension and revocation of authority may also be driven bythe content of the messaging session itself. If the virtual assistantsystem 10 is managing a messaging session, the client 110 could also bemonitoring the content of the messaging for relevant commands, which maybe forwarded to the assistant server 200 to carry out the associatedactions. For example, a user in a messaging session might ask permissionfrom another user to use a particular service 300 a, e.g., “can I usethe dropcam?”, and this question could be translated into one or moresemantic atoms 1000 by the assistant interface 110 and sent to theassistant server 200. The assistant server 200 could interpret thisstatement as a request to share access to the relevant service 300 a, inthis case the dropcam. If the owner of this service 300 a responds withan unambiguous reply such as “sure,” this reply would similarly be sentas one or more semantic atoms 1000 from the owner's assistant interface110 to the assistant server 200, the assistant server 200 wouldunderstand this reply as granting the request, and accordingly authorityfor the service 300 a can be automatically extended to the requestinguser, without the need for a separate operation by either user.Similarly, if one of the users in the chat session asks “are you doneusing the dropcam?” and the user who was granted access repliesaffirmatively, e.g., “Yes, I am,” the assistant server 200 canautomatically revoke the access which was shared. Since sharing can beperformed automatically based on the content of the messaging session,users do not need to turn away (revert to another form of control) frommessaging in order to separately control the virtual assistant system 10to share access. In this manner, sharing access and control of services300 may become a more seamless part of messaging.

Incorrect granting and revocation can also be addressed by the adaptivelearning approach described above. If the user revokes authority as soonas such authority is granted, for instance, that is an indication to thevirtual assistant system 10 that it overstepped in granting thatauthority.

This approach, of actions taken based on the content of messaging, canbe extended to transactions as well. For example, if a first and asecond user discuss watching a movie at a particular theatre, and thefirst user asks the second user “I have found the tickets but can youpay for them?” this question may be converted to one or more semanticatoms 1000 by the first user's assistant interface 110 and sent to theassistant server 200, which may interpret the question as a request forpayment for a specific purpose (payment for tickets in this case). Ifthe second user assents, for example by responding “sure,” a selectedpayment instrument of the second user may be applied to the specificpurpose, in this case purchasing the tickets. The assistant server 200interprets the second user's response as agreement to the request of thefirst user, and therefore extends authority to the first user for theselected payment instrument for that amount of money and that specificpurpose (buying the tickets).

This use of the messaging content as command inputs may be performedautomatically or through conscious user interaction with the assistantserver 200. In the examples given above, the assistant interface 110included in the mobile device 100 may automatically translate perceivedcommands (e.g., questions and responses from the users) into one or moresemantic atoms 1000 and forward the semantic atom(s) 1000 to theassistant server 200, which may automatically take the correspondingaction. Alternatively, or in addition to such automatic responses, theusers may send commands to the assistant server 200 through the mediumof the messaging session. In this situation, the assistant server 200is, in effect, a “participant” in the messaging session in that theassistant server 200 may receive messages (corresponding to commandsfrom users) and also transmit messages (e.g. replies, outputs fromservices 300, etc.) within the messaging session. Multiple assistantservers 200 a-n in communication with each other, constituting multipleinstances of the assistant server 200, may also be participants in amessaging session, similarly to how services 300 a-n may participate inmessaging sessions.

Referring to FIG. 18, a user of the mobile device 100 can add, includinga mass-addition, services to an assistant server 200 based on aquestionnaire, survey or form-filling of credentials and otherservice-related data, as well as based on discovery.

More specifically, it is likely that when the assistant server 200becomes established other users will want to provide all theirindividual account credentials to the assistant server 200 so the otherusers can rapidly establish connections to those services 300 a-n. Forexample, a user can give his/her usernames and passwords to LinkedIn™,Yahoo™, Google™ Amazon™, Facebook™, etc., to the assistant server 200 atone sitting, which enables the assistant server 200 to systematicallyconnect to these services 300 a-n all quickly. This can be done byfilling in a form that the assistant server 200 provides to the users.This form can be a spreadsheet, email form, fillable PDF, web-basedform, a voice-based question and response session, or any other datacollecting mechanism through which this type of information can becollected. The form can be delivered to the user via the Internet or viasome other connection mechanism such as Bluetooth or NFC, or evenphysical mail or courier service.

The assistant server 200 can also take the user through a process wherethe user is asked for credentials to most common services 300 a-n thatthe user is likely to be using—which is similar to filling out a survey.For example, the assistant server 200 can detect services 300 a-n on theuser's home network and then connect to them automatically by requestingthe corresponding credentials when necessary. Information relating tothese credentials and services 300 a-n can be saved on a storage 250that is connected to the assistant server 200.

Similarly, the assistant server 200 can actually take the user's emailsand then probe most common services 300 a-n with those emails to see ifthat user already has associated accounts. In this case, the user canprovide various email addresses to the assistant server 200, which cansave these email addresses and then use them to attempt to log intovarious web services and other services. If these users have forgottentheir passwords, the assistant server 200 can initiate a password resetprocess (i.e., either by conducting the password reset processautomatically or with the user's help if services like Recaptcha™ areinvolved, which require humans to recognize obfuscated letters andnumbers) and revive those accounts and link them to the user's accounts.This also means that users may no longer have to come up with passwords.The assistant server 200 can come up with highly complex passwords whichthe user does not need to remember since the users are almost alwaysgoing to access the underlying services 300 a-n through the assistantserver 200. These highly complex passwords may then be saved on thestorage 250.

The assistant server 200 can also take a user through a process ofestablishing brand new accounts with various services 300 a-n that arerelevant to people similar to the user. This can have many applications.For example, the assistant server 200 can be used to establish a votingaccount in an election system for a citizen when they achieve a certainage. It can be used to enable the citizen to establish an account withbmw.com when they buy a BMW. The rules determining such new servicesetups can either be saved on the storage 250 or be communicated to theassistant server 200 by other services 300 a-n that are able tocommunicate with the assistant server 200 over the Internet, WiFi,Bluetooth, NFC, USB or other mechanisms.

The virtual assistant system 10 can also facilitate connecting services300 a-n to each other. For example, if a user buys a video game system,the user may wish to connect it to a television. In this case,conventionally the user must use cables to perform an analog connectionbetween the video game system and the television. Alternatively, if boththe television and the video game system have wireless capabilities, theuser has a wireless router, and the user is subscribed to an InternetService Provider, then the user can wirelessly connect the video gamesystem to the television by searching for the video game system on amenu screen of the television, for example, and then selecting the videogame system to connect wirelessly to the television. This procedure mayrequire searching through multiple menu screens of the television tolocate the video game system on a user's home network, as well aspotentially requiring the user to have knowledge ofconfiguration/connectivity specifications of the devices that are to beinterconnected. As such, the conventional procedures to connect twoservices (the video game system and the television in this example)cause inconvenience for the user.

However, the virtual assistant system 10 can provide the aboveinterconnectivity without requiring the user to have an Internet ServiceProvider, a wireless router, or knowledge of configuration/connectivityspecifications between the services. Also, the virtual assistant system10 avoids the need for the user to search through multiple menu screensto find the desired service to perform the interconnection.

More specifically, when a user acquires a new service 300 a, for exampleby purchasing a new device, plugin 220 a corresponding to the service300 a can instantly be installed on the user's mobile device 100, oralternatively in the assistant server 200. This installation can occurmanually by the user downloading the plugin 220 a, e.g., from theInternet, and installing the plugin 220 a on the mobile device 100 orassistant server 200. Alternatively, the plugin 220 a can be sent viaemail, text message, another user's virtual assistant system 10, or anyother type of communication method. For example, when purchasing the newservice 300 a, the user could simply provide an email address to themanufacturer for registration, which would in return automatically sendthe user the plugin 220 a corresponding to the service 300. Furthermore,if the mobile device 100 and/or assistant server 200 is instructed whereto find the plugin 220 a (i.e., where the plugin 220 a is stored), themobile device 100 and/or assistant server 200 may retrieve and installthe plugin 220 a without the user needing to separately send the plugin220 a to the virtual assistant system 10.

Then, when the plugin 220 a is installed in the mobile device 100 orassistant server 200, an icon representing the new service 300 a (e.g.,a video game system) may appear, for example on the user interface 150of the mobile device 100. The icon could automatically appearspecifically in a “Services” menu screen of the mobile device 100, forexample.

In order to connect the new service 300 a to another service 300 b (forexample, a television, which may already be represented within themobile device 100, for example by an icon installed therein), the usermay make the relevant command input, which may be intuitive, and requireno detailed knowledge of connectivity. This command input could be anytype of action that would make it appear that the services 300 a and 300b are linked. For example, the user could simply drag an iconcorresponding to a video game system to an icon representing atelevision on the user interface 150 of the mobile device 100. Thevirtual assistant system 10 may interpret this command input as acommand to connect the two services 300 a and 300 b. In this example,the video game system would be wirelessly connected to the television,so that communication could be performed wirelessly therebetween. Morespecifically, since the video game system and the television bothcommunicate in their own proprietary languages, installation of plugins220 a and 220 b respectively corresponding to the video game system(service 300 a) and the television (service 300 b) allows thecorresponding proprietary languages of the services 300 a and 300 b tobe translated into semantic atoms 1000 via the virtual assistant system10, which allows the services 300 a and 300 b to communicate with eachother.

The above procedure can be used to connect any number of services 300a-n that require communication therebetween. For instance, a televisioncan also be connected with external wireless speakers and an externalwireless Blu-Ray player. Thus, the need for detailed “set-up” operationsto enable services 300 a-n to communicate with each other is eliminated.

The present general inventive concept can also be embodied ascomputer-readable codes on a computer-readable medium. Thecomputer-readable medium can include a computer-readable recordingmedium and a computer-readable transmission medium. Thecomputer-readable recording medium is any data storage device that canstore data as a program which can be thereafter read by a computersystem. Examples of the computer-readable recording medium include asemiconductor memory, a read-only memory (ROM), a random-access memory(RAM), a USB memory, a memory card, a blue-ray disc, CD-ROMs, magnetictapes, floppy disks, and optical data storage devices. Thecomputer-readable recording medium can also be distributed over networkcoupled computer systems so that the computer-readable code is storedand executed in a distributed fashion. The computer-readabletransmission medium can transmit carrier waves or signals (e.g., wiredor wireless data transmission through the Internet). Also, functionalprograms, codes, and code segments to accomplish the present generalinventive concept can be easily construed by programmers skilled in theart to which the present general inventive concept pertains.

Although a few embodiments of the present general inventive concept havebeen shown and described, it will be appreciated by those skilled in theart that changes may be made in these embodiments without departing fromthe principles and spirit of the general inventive concept, the scope ofwhich is defined in the appended claims and their equivalents.

1. A mobile device for use in presenting a virtual environment to auser, the mobile device comprising: a sensor configured to receive aninput command from the user, the input command corresponding to afunction to be performed at one or more external services; an assistantinterface configured to translate the input command into a semantic atomrepresenting the input command; and a communicator configured totransmit the semantic atom to at least one other user of the virtualenvironment.
 2. The mobile device of claim 1, the virtual environmentbeing presented as part of a virtual reality (VR) session amongst aplurality of participants.
 3. The mobile device of claim 1, the virtualenvironment being presented as part of an augmented reality (AR) sessionamongst a plurality of participants.
 4. The mobile device of claim 3,further configured to receive a semantic atom representing output datafrom the AR session, and to present the output data to the user of themobile device.
 5. The mobile device of claim 1, wherein the sensorcomprises a stationary sensor.
 6. The mobile device of claim 1, whereinthe sensor comprises a wearable sensor worn by the user.
 7. The mobiledevice of claim 3, wherein at least one of the plurality of participantsincludes at least one of a plugin, another mobile device, an assistantserver, and another user.
 8. The mobile device of claim 3, wherein aparticipant in the AR session transmits one or more semantic atoms toone or more other participants in the AR session.
 9. The mobile deviceof claim 7, wherein a first plugin participating in the AR sessiontransmits a semantic atom representing at least one of settings, userpreferences, user status, user credentials, and command inputs to asecond plugin participating in the AR session.
 10. The mobile device ofclaim 9, wherein the first plugin transmits the semantic atomrepresenting at least one of settings, user preferences, user status,user credentials, and command inputs to the second plugin automaticallywhen the user inputs a command to the second plugin.
 11. The mobiledevice of claim 1, further configured to analyze context information ofthe input command before the input command is converted to a proprietarylanguage command to accurately determine the intended function to beperformed by the input command.
 12. The mobile device of claim 11,wherein the analyzed context information includes at least one of whichplugin is receiving the input command, a status of a participant in anAR session, a time of day when the command is input, co-presence ofspecific individuals, involuntary actions by a user, and actionsequences taken by at least one user.
 13. The mobile device of claim 12,wherein the action sequences taken by the at least one user includes aplurality of input commands made simultaneously or sequentially.
 14. Themobile device of claim 12, wherein the action sequences taken by the atleast one user includes an input command that is invalidated andfollowed by another input command.
 15. The mobile device of claim 12,wherein the action sequences taken by the at least one user includes aninput command that is made by a first user and another input commandthat is made by a second user.
 16. The mobile device of claim 12,further configured to update a stored library of recognizable inputcommands according to the analyzed context information.
 17. The mobiledevice of claim 12, further configured to update a stored library ofrecognizable input commands corresponding to a first AR external serviceaccording to analyzed context information corresponding to a second ARexternal service.
 18. The mobile device of claim 13, wherein when themobile device recognizes a repeated cluster of input commands, themobile device creates a new consolidated input command to representperforming the cluster of input commands simultaneously and stores theconsolidated input command in a library of recognizable input commands.22. The mobile device of claim 3, further configured to storecredentials including data to authorize one or more participants of theAR session to issue commands for the one or more AR external services.23. The mobile device of claim 22, wherein: at least one participant ofthe AR session controls an assistant server to extend authority to inputcommands to the one or more AR external services to at least one otherparticipant of the AR session.
 24. The mobile device of claim 23,wherein the at least one participant of the AR session sets a level ofauthority the at least one other participant of the AR session has toinput commands to the one or more AR external services.
 25. The mobiledevice of claim 22, further configured to automatically authorize one ormore participants of the AR session to control the one or more ARexternal services through input commands according to the storedcredentials, the authorization being based on context information. 26.The mobile device of claim 22, wherein a plurality of the participantsof the AR session controls the assistant server to extend authority toinput commands to the corresponding AR external service to each other.27. The mobile device of claim 22, wherein: the stored credentialsinclude at least one payment credential; and at least one participant ofthe AR sessions controls the assistant server to extend authority to usethe at least one payment credential to at least one other participant ofthe AR session.
 28. A method for presenting a virtual environment to auser, the method comprising: receiving an input command from the userthrough a sensor, the input command corresponding to a function to beperformed at one or more external services; translating the inputcommand into a semantic atom representing the input command; andtransmitting the semantic atom to at least one other user of the virtualenvironment.
 29. The method of claim 28, the virtual environment beingpresented as part of a virtual reality (VR) session amongst a pluralityof participants.
 30. The mobile device of claim 28, the virtualenvironment being presented as part of an augmented reality (AR) sessionamongst a plurality of participants.